It’s the only explanation!

Related

Ace has dealt with problems before…

Default blame accepter?

The post Changes (Ace Comic) appeared first on Steve Karam :: The Oracle Alchemist.

All work and no play makes us dull DBAs.

Being a DBA can be a really thankless job. From Jr. Shutdown Immediaters to Sr. Rockstar Oracle DBA ACE OCM PMP LOL A+ BBQs, we’ve all trudged into work at some point in our careers only to find that we have three impossible tasks, four humiliating tasks, and no less than seven “why is the database causing this query to be slow we haven’t changed it in two years” tasks. Yet we keep on doing it, day after day, for love of the data. Or something like that.

I wrote an article a while back about galvanizing the DBA team into action and making them stand out a bit more in the corporate world. In the end, a lot of that boils down to making sure the DBA team looks less like a cranky support group that constantly needs disk space and expensive Oracle licenses and more like a team that wants to work as a key part of bringing new products to market (and thus earning money). But even then, it can be a tedious, mind numbing task at times.

So how can we spice up the DBA experience and the IT life in general?

Gamification, of course

Everybody loves games. More to the point, people like being competitive and reveling in a sense of accomplishment and achievement. The idea behind gamification is that many things in the day-to-day non-gamified world can have principles of game theory attached to them. For the most part, the current trend of gamification is a corporate strategy that can be used to further engage a customer base (and is predicted to be a $1.6bn market by 2016 as a result). But even from a departmental or team point of view, it can have great benefits to increase morale, productivity, and accomplishment. And with a little luck and project management, it can be used to solve critical issues for your business.

What gamification is:

• Making tasks more like missions. Everything from tuning a query to backing up a database can be a mission. Instead of becoming yet another ‘requirement’, tasks can become goals.
• Providing a system of Achievements, Badges, Rewards, etc. Modern games are nothing if they don’t have some sort of achievement system that allows players to share their accomplishments across their social network or game community. Within a business, that community can be the IT group (like the DBA team), the department as a whole, or even business-wide.
• Competition! Pitting people against each other in a fun, non-threatening way is a great way to improve morale and encourage growth of skills outside of the standard “do this and do it well or you won’t get a raise” paradigm.
• Leaderboards, which are good for tracking the competition and achievements but are also useful for status and review purposes. What’s the point of earning achievements if they aren’t tracked, or competing without scores?

What gamification is not:

• Strapping your DBA team into slingshots and giving them an achievement for knocking over the most developers.
• An IT version of Thunderdome where two DBAs enter, and one DBA leaves.

Let the Games Begin

So it sounds great in theory, but what can we possibly do to bring it into our DBA practices? As with all answers regarding database management, it depends. If you’re a consulting organization, gamifying is simple because you generally are performing work and solving problems for a wide array of clients. You track team members’ status and customer satisfaction anyways because it is part of the business model. For a DBA team inside an IT department at a major corporation, it can become a little more difficult. So let’s look into some of the ways you can make games part of your actual work.

Backup and Restore Challenges

Setting up and monitoring a good backup and disaster recovery solution can be a serious pain in the neck. But everyone would agree it is absolutely vital to your company and your job security. But even worse than setting it up and monitoring it is having to actually put those backups to use in the case of a critical failure. Whether the problem is a single corrupt data block or an entire database ground to dust, having to actually perform a restore when the heat is on and the managers are circling and all your plans for the weekend are being piled up in the “it’s the thought that counts” bin like so much garbage… well, it sucks. There, I said it.

Maybe that (with a hint of staffing problems and financial woes) is why companies are failing to adequately test their DR plans. That’s a highly dangerous practice, right there with giving developers DBA access to production and typing drop database commands and saying just kidding while you fumble for the backspace key. What is the point of organizing and configuring a complex backup solution if you only plan on testing it when disaster strikes? I can’t count the number of times I’ve worked with clients who said “oh, we needed that file?” or “well, the logs all looked okay” as their hopes for database recovery dropped like their company’s stock price after a three day outage.

So test your backup and recovery plan. But you can make it more fun by running simultaneous tests or presenting interesting and sometimes horrifying cases to the team when they are asked to perform restores. In this article I wrote about the Backup/Restore Workshop day when I worked for Oracle University, and how one student took his preparations to a whole new level, ultimately giving me the old what for, so to speak. Make a competition out of it. Challenge your DBAs to recover from sticky situations and to do it quickly and efficiently. Engage other teams too (such as application server administrators), who ultimately should be part of backup and recovery testing. Doing so invites the teams to improve their speed while also giving the business a good idea of where there may be bottlenecks.

Questionable Query Quality

Not only is this topic worth 192 points in a triple word score in Words with Friends, it’s also a serious issue that almost every DBA/Development group has to face. Writing queries can be a very tedious task to the point where many groups skip that whole tuning thing and go straight for production, guns blazing. This is, of course, a bad idea.

At one company I worked for we came up with a really interesting, fun, and productive way of tackling this issue: we pitted the query developers against each other in battle. Developers would requested to put a comment in their SQL code with their name so we could 1) track who wrote it, and 2) hold contests periodically. So when someone did “SELECT /* Steve Karam */ count(*) from RIDUCLOUSLYHUGETABLE;” we knew who it was and where to go to get it fixed.

That can make for some seriously awesome competition. Each month, quarter, release, etc. you can host ‘award ceremonies’ honoring the best and worst of the time window. Grab explain plans for all the queries and come up with valuable metrics to track like:

• Most readable
• Best use of indexes
• Least blocks touched
• Most convoluted query
• Least likely to make it to production
• Most likely to result in concurrency waits

It’s all in good fun. And it makes query reviews almost bearable. In fact, I’ve noticed in cases where I used this (whether in real environments or in training sessions) the participants were not embarrassed or angry about writing a bad query. They wanted to figure out how to make it better so they could be in a different acknowledgement tier the next time around.

Professional Growth Challenge

While there are a ton of competition ideas within the DBA team (best use of indexes, most SQL profiles used, fastest mouseslinger in the Grid) there is also that of professional development. How about a competition on who can get their OCP certification first? Or one of the many Oracle Expert certifications out there? Heck, even a contest to see who can rack up the most obscure or highest count of certifications from anywhere in the industry would be fun and potentially enlightening. Not only will the team benefit from the certifications and the process to get them, but it makes the team look better as a whole when everyone is clamoring to get professionally acknowledged the fastest.

If you have a really top notch team, then challenge them to get their OCM. While this will require some company backing to keep it fair since the OCM is an expensive and time consuming test to take, it definitely has its rewards. It is a great way to encourage professional growth within your team using pre-built metrics.

D&D Tabletop Drills

My current company had a pretty cool idea as the result of a stability and process review initiative. We started doing tabletop drills once per quarter. For these drills we assembled a team of moderators (usually one manager from each IT group) that came up with a couple scenarios of problems that can occur in our production systems. A ‘player’ is chosen from each team to act as representative: one DBA, one applications administrator, one system administrator, etc. And we would get everyone in a room, surrounded by observers, and act it out.

The idea is to act out the response to the crisis exactly as we would if it were really happening. Sometimes input is needed; for instance, if they discover that this scenario includes application servers being down, they can ask how many are down. Moderators answer questions and keep track of the progress to gauge how the activity goes as a whole, whether SLAs were acknowledged and met, and what can be improved. Following the activity, observers, moderators, and participants have a discussion about the scenarios as a whole and what action items should occur as a result of the findings.

I like it because it’s like the IT version of Dungeons and Dragons. We’ve got game masters and a group going through a perilous journey through a broken and charred land that was once a shining beacon of production goodness. Heck, we could even include dice next time. “How many DBs are down?” “Roll 2d6″. Critical hit, go to DR!

What’s In It For Me?

Even though there are built-in rewards to many challenges — better queries, faster backups, certifications, what have you — people still want to have something a little more, particularly in the gamified experience. So what can you offer up your team or colleagues for a job well done?

Achievements: One of the cornerstones of game theory, achievements are fun and fulfilling. Use stickers, logos, Portal shoutouts, even badges inside the company intranet to show accomplishments. I personally think it would be neat to see “Achievements” next to Job Description, Contact Information, and Org Chart in my company portal filled in with things like “Onward and Upward – Upgraded three databases in a two week period” or “Did I Do That? – Fixed a fat-finger mistake with flashback or restores”. In fact, I would go out of my way to do amazing things just to earn more achievements.

Seminars and Conferences: Maybe the DBA with the most accomplishments gets to go to Oracle Open World this year. Or perhaps the developers with the best (or worst, to be more logical) queries get tickets to the next big Java conference. Whatever the case may be, I’ve noticed a growing number of companies unwilling to send their teams to conferences. In some cases they believe them to be merely a chance for the DBA (or other professional) to find another job. In others, they see it as a waste of money compared to other training. While ultimately these are horrible reasons to deny technical conferences to your team, perhaps using achievements and fulfillment of goals as a way to send people will improve performance as well.

A break: You could always give the winner a day to work from home (or off, you taskmaster you). Or for the thrill seeking types, the ‘losers’ have to give the winner a week off on call by taking it for them.

Will Turner: What are they wagering?

‘Bootstrap’ Bill Turner: Oh, the only thing we have. Years of service.

Company Certifications: Why not create a set of certifications within your own organization? As someone who has been in education and publishing for quite a while and as one of the lead writers of the Oracle 10g Certified Master exam, you can trust me when I say that certifications are an outstanding way to bring job satisfaction and a feeling of recognition to an IT team. Come up with certifications within your company that can be awarded for exemplary work or completion of certain achievements. Use those certifications during review time as a concrete measurement of success within the company. Make it possible for anyone to get certified in-house (e.g. developers can get in-house DBA certifications) to keep the career paths open at your organization. There are many ways you can acknowledge people for hard work; a certification they can put on their company profile, their emails, or even their resume can go a long way toward that goal.

Let them destroy something: This one sounds funny, and it is… but next time you migrate from one server to another or decommission an environment, let the winners of various competitions be the ones to type and press enter on that DROP USER command or DROP DATABASE. A company I worked for a long time ago did this. We finally moved off of an old and painful architecture. Once the database was moved over and the old one set to be deleted, we allows developers to drop the tables they hated the most and gave the honor of dropping the main schema to our lead (and most overworked) customer service representative. It was a great way to celebrate the occasion.

Conclusion

Some of these ideas may seem far fetched or downright silly. But you have to track employee progress, tasks, and status anyways. Why not do it in a way that gives everyone a sense of competition and accomplishment?

Gamification doesn’t have to be strictly for companies to rope in consumers. By using it within teams it can also improve morale, grow employees at a professional level, and give everyone something to strive for.

The post Gamifying the DBA Experience appeared first on Steve Karam :: The Oracle Alchemist.

More Friday fun from the World’s Leading DBA Comic!

Related:

Metrics That Matter: These Aren’t the Metrics You’re Looking For

Improve Your Experience: DBA, Grow Thyself – Moving and Shaking in the Era of Data Dominance

The post Industry Leader (Ace Comic) appeared first on Steve Karam :: The Oracle Alchemist.

April is almost over, and you know what that means…

Oracle really, REALLY wants to talk to you about some of their products.

It’s no secret that the end of Oracle’s fiscal year (May 31st) marks a flurry of activity for Oracle sales, and doubtless many of you have met or will be meeting with them soon. These meetings can be highly beneficial for your company particularly if you are on the hunt for new hardware, database licenses, or other software. And if you’re not on the hunt for anything new, well… they can be pretty persuasive. You might be the proud owner of a T5-8 on June 1st.

Before that time comes, there’s a few things you can do to prepare. Ready yourself!

Understanding Licenses

Oracle keeps pretty good records. Before meeting you they will make sure to know exactly what you own, how much you pay for support, and what your company’s public plans might be. They absolutely understand their own licensing model and know a good deal about how it plays into your environment.

So why would you go into a discussion with them about purchases without knowing what it is you’re actually buying?

First, make sure you understand the different license types and how they apply to your servers. Know whether you need Standard Edition or Enterprise Edition. Perhaps you are concerned about missing out on Diagnostics Pack and AWR/ASH. Yet perhaps STATSPACK and Simulated Ash (S-ASH) might work for you instead. Do you need DataGuard? Try DBvisit. Make sure before you take the Enterprise Edition plunge you really need all the features and add-ons it allows.

Don’t get me wrong, those features are many and wondrous. Parallelism, online operations, DataGuard, Tablespace Point In Time Recovery (TSPITR) and more are features worth having… if you want to pay for them.

Get a Grip on Your Licenses

It is equally important to understand your own licenses. The many companies I’ve worked with were a mixed bag. Some paid extraordinarily meticulous attention to their licenses, either entrusting them to an employee in charge of corporate license purchases or to the DBA team. Others had absolutely no idea and no control over their licenses. Developers, DBAs, or even small groups working under non-IT departments would install Oracle on an as-needed basis with no care for the complicated requirements of Oracle. Every few years they’d scramble out of fear of the audit, that oft whispered pain that may rain down on an unruly business.

I’ve never been through an audit so I can’t speak to their supposed rigorousness. But I do know the best thing you can possibly do — both to stay in the good graces of Oracle and to know what you need come bargaining time — is to know where Oracle is being used inside your organization and what features you have installed.

This might include a company-wide internal audit of your installations, asking each group what they have installed and where. A DBA team that is not aware of ‘secret’ Oracle installations is in for quite a bit of pain anyways, as when problems strike you will suddenly become aware of it in the form of a team of people requiring help. It is also recommended that you run scripts on your installations to gather information about installed features from DBA_REGISTRY. Make a spreadsheet, fill it in, and keep it updated. Track the licenses you’ve bought and where they are allocated. And if you are out of compliance, make note of it. The end of Oracle’s fiscal year may be a great time to true up.

Pay Attention to Metrics that Matter

I wrote an article recently about the misunderstanding behind some Oracle metrics called These Aren’t the Metrics You’re Looking For. In it I described five frequently misused metrics as they pertain to Oracle tuning.

But there’s another very very misunderstood metric as most people don’t even know it exists or why it’s important. And that is the only metric that matters when it comes to hardware purchases for database servers: per-core performance.

Why is this important? Because as Kevin Closson points out in his excellent article about the SPARC T5 Processor, performance per core can be correlated to performance per license. If you’re about to shell out your hard earned cash on a fistful of licenses then you need to know exactly what each license buys you. If Oracle’s licenses were based on the amount of disk you used, you would absolutely be sure to make the best use of that disk.

I’m not saying that the other things Oracle will bring up — caches, disks, secret sauce, oh my — are not important. Depending on your bottleneck the juice may be worth the squeeze. But before you buy a server with oodles of CPUs that you must license perhaps you should make sure your application is tuned to make the best use of CPU. Then perhaps you should make sure that server meets your CPU needs.

Do You Even Need Anything New?

Why are you sitting at the table in the first place? Is it because you really need to make some purchases or because you got a call? Or perhaps because someone up above made the request.

There’s been a long standing debate about throwing hardware at a problem. More and more companies these days are throwing hardware at issues… either to avoid expensive and time-consuming tuning and re-architecting or because standard computing can’t do the things they want to do. I’m not going to embark on that particular journey in this post, but I will say that you owe it to yourself and your finances to at least ensure your problems can’t be resolved with some parameter changes, query rewrites, or indexes.

Absolutely most importantly, make sure the hardware you are buying fixes the problem you are having. I’ve seen many companies buy new hardware only to find it didn’t make a single difference in their performance. And there is nothing that will anger a business and frustrate an IT department more. Pay attention to waits and usage, and make sure the bottlenecks you’re encountering are the ones you’re fixing. If your system is disk I/O bound, a new microprocessor probably won’t give you much, if any, relief.

Watch Out for the Eager Want

A principle of sales from my favorite business book, How to Win Friends and Influence People, is to “arouse in the other person an eager want.” Remember that Oracle’s “eager want” is already there. They want to sell you products all year round, and definitely right before the end of their fiscal year. When Oracle Sales comes knocking at your door, their job is to arouse that want in you.

You will hear about all sorts of amazing products, from SPARC chips to OVM to Advanced Compression to Real Application Clusters. You don’t have to wear a hat and shades to hide your emotions like a poker competition, but you may at least want to make sure you know what these features do and whether or not you need them. It is good to have a healthy relationship with Oracle. They are a maker of outstanding products, and they absolutely have knowledge of those products that will be beneficial to you. But just like with any sales meeting, make sure you keep your wants in check and mind the pocketbook.

Conclusion

Twas the month before end of fiscal year and all through the net, sales are stirring and getting us in debt. This is absolutely the time to get some outstanding deals from Oracle Corporation (and other companies). Get your ducks in a row and reap the benefits responsibly!

The post Bringing Oracle Sales to the Table appeared first on Steve Karam :: The Oracle Alchemist.

A couple weeks ago I posted a whimsical (and very slightly sardonic) article on the Internet of Things with a view on what an average day could be like if everything (even your shower soap tray) was network connected and capable of programmed rules or even decision making analytics.

It spurted a generous number of small Twitter conversations and a spot on a couple Internet of Things aggregators. But just a couple days ago I saw an interesting tweet:

Talking about “Does the internet of things create a future without surprise” on @branch. Who has something to add? branch.com/b/does-the-int…

— Matthew Knight (@webponce) April 18, 2013

Introducing Branch

I had heard of Branch before, and I had understood it as a place where you can have conversations about a topic with more than 140 characters. One of the things I noticed was that I was unable to initially reply to the branch conversation; instead, I had to ask the original conversation author to invite me. It was described to me by @branch thus:

@oraclealchemist The former. Gives you the ability to host little “dinner table” conversations, if you want to think about it that way.

— Branch (@branch) April 18, 2013

It turns out the reasoning for this is pretty cool. Have you ever been involved in a conversation on Twitter and wanted to take it offline? That’s where Branch comes in. It’s still in the public eye, but now you get 750 characters per response. The people who were on the Twitter conversation with you are still part of the Branch conversation, and other people can ask to be involved. A dinner chat indeed! You can also share any point in the conversation back to Twitter, even a single sentence of someone’s contribution. And the conversation as a whole can be embedded into any site. Here, for example, is the one regarding my Internet of Things article:

It is a fairly cool platform, and I could see it being incredibly useful in the IT world thanks to our long product names, syntax, and tendency to talk too much (okay, maybe I’m projecting).

The post Branch Off a Twitter Conversation appeared first on Steve Karam :: The Oracle Alchemist.

In my adventures in Twitterland I’ve noticed a lot of people who are using WordPress blogs (this one included). Some use wordpress.com which gives them a free blog with minimal theme and plugin choices; however, they get a lot of cool optimizations built in.

If you’re like me and running your own self-hosted blog then you have all the options in the world available to you but far less right out of the box.

This post will go over the performance changes I’ve made in an attempt to get my blog up to speed. It is part one of a series called Optimizing WordPress which will cover performance, syndication, editing, and more.

For now, let’s drop a gear and disappear speed up our blogs!

WordPress – Location, Location, Location

The first major point of consideration is this: Are you master of your own fate, beholden to no one, or a slave to the will of others? And by that of course I mean: do you have root?

I wholeheartedly recommend having root privileges if you are going to run your own blog (particularly if you know what you are doing). Shared hosting of the bygone age generally did not allow this, but today you can get yourself setup via a number of dedicated or cloud virtual private server (VPS) environments and have your own fully configurable server in no time flat. I personally host mine with 15 Minute Servers as I want a robust environment I can use for many of my own sites, shared hosting for friends, etc.

Some providers are actually dedicated strictly to WordPress such as the ever popular WP Engine. These options will cost a bit more than running it yourself, but they also come complete with the enhancements you will probably want.

Whatever you choose to do, get root privileges. Not only will it make you feel lordly, but it will be necessary for the next step of this article.

Let’s Get Cached

There’s a piece of software out there that does such a good job at providing caching services to PHP that PHP will include it in the core sometime soon. Until then, I’d recommend you download Alternative PHP Cache (APC).

With APC installed PHP code can now take direct advantage of the RAM resources on your system and use it for caching chunks of data. MySQL results, objects, and even entire HTML pages can be cached inside of it with the right code.

Alone this cache won’t do much for you, but with the right plugins for WordPress you’ll be able to take full advantage of it. Before that point though we must get the APC software installed. You can find a good guide for that here.

And as if the software benefits weren’t enough, you also get a great monitoring page with all sorts of metrics to keep track of which I know my Oracle buddies will absolutely love.

Cold Pressed Performance

Once you have APC installed, you can go ahead and install a plugin in WordPress called W3 Total Cache. This is an extremely well known plugin that is able to cache various parts of the WordPress environment, including:

• Pages – When a user reads a page it will be cached so the next hit will be near instant.
• Minify – There are two parts to this component. It will take your JavaScript/CSS files and eliminate whitespace and change variable names to be smaller. It will also cache these ‘minified’ files for better performance.
• Database – Caches MySQL query results. Highly important for WordPress, where depending on your theme you may be running over 100 queries for a single page.
• Objects – Caches WordPress cache objects into your chosen cache (in our case, APC).
• Browser – Allows browser-based caching on the client side.

Turning these on is as simple as installing W3 Total Cache, navigating to the Performance > General Settings page, and selecting these caches with the Cache Methods set to “Opcode: Alternative PHP Cache (APC)”. There are other settings that may interest you which you can find under each cache type’s individual page under the Performance section of your admin page. One incredibly interesting option is the CDN, which stands for Content Delivery Network.

It’s not Delivery, It’s CDN

Content Delivery Networks are the coolest thing since ever. Simply by setting up a CDN with a “pull zone” and fiddling with some DNS settings you can put the power of many servers at your disposal.

CDNs are designed to take your static content such as images, CSS files, and JS files and distribute them over a wide network. This distribution is designed to be automatic both for you and your end user. Viewers on your website see nothing but sudden blazing fast speeds as the images and code for your pages are delivered to their browser from a variety of sources.

There are many CDNs to choose from. CloudFlare is an extremely popular one due to its performance and security features while also being somewhat easy to set up. I went with MaxCDN due to reviews I had read and the customizability it seemed to offer and I am very happy with it. Setup can be a little harrowing; basically you need to create an account, fork over some cache, and add CNAME records on your DNS pointing a subdomain to your new MaxCDN account. On the MaxCDN side, you will configure a Pull Zone which is configured to pull content from your blog on request and serve it up. There are options to clear cache files and all sorts of cool reports and other features.

There are a ton of settings to use and they are well documented. Instead of going through the entire thing I will point you to How To Install and Setup W3 Total Cache For Beginners. Just remember that we are using APC for caching, not Disk. The most important part of the article at this point is the MaxCDN and W3 setup for it.

Once you have Page, MySQL, Object, and Browser caching enabled along with a CDN to serve up your images, JavaScript, and CSS from all over the world you will be, for the most part, as fast as a software company jumping on the Cloud bandwagon.

One item of note: There’s a setting under Performance > Page Cache to enable/disable the cache for either logged in users or certain user roles. It is highly recommended that you turn off page caching for Administrators. That way you can see the results of your hard won changes. The only drawback is that your users will have faster performance than you, and may notice problems that you don’t. It’s always a good idea to have another browser ready that’s not logged in to your admin account to keep track of pages and how they look.

What’s Slowing You Down?

Even with all these optimizations you still may have some pain points. A lot of times these are caused by plugins that are simply too overbearing on user performance. Many plugins have components that ‘phone home’ to the plugin owner’s site for whatever reason which can drastically reduce your performance.

The plugin P3 (Plugin Performance Profiler) is able to review your site and give you great details on which plugins are causing the most pain. You can find the plugin here.

Once you’ve installed the plugin, direct it to the pages you’d like to test and let it go to work. It takes a while but gives you table and graph data to let you know what you need to disable, replace, or tweak.

Honestly the best policy overall is to review your plugins periodically. Many times you’ll find that there are a ton of plugins you just don’t need, or maybe stopped needing recently.

Testing, Testing 1ms 2ms 3ms

While you’re doing all these performance improvements it’s a good idea to keep an eye on your page speeds to track your progress. Pingdom Tools has a great speed test that can test your site from multiple locations and let you know how long it took. You also get a breakdown of where the performance hogs were so you know what to tune next (for instance, a large javascript file or big image). Just because you cache and distribute content doesn’t mean you will be fast for all users; remember, they still have to download the page which takes time. The more jQuery and reactive design stuff you have on your page, the bigger it is going to be.

Another good one is WebPageTest which I actually like quite a bit better than Pingdom. It gives you greater options for how you want the test conducted and can save the results so you can compare later. However, both are capable and can give you great clues as to which files are causing you grief.

What Else?

That’s the end of this performance boosting article, but there is more to come. Next in my Optimizing WordPress series is syndication, where I will go over different syndication and sharing plugins and technologies available to you to help get your blog to more places.

The post Performance Tuning Your WordPress Blog appeared first on Steve Karam :: The Oracle Alchemist.

Related:

Loosely based on my (tongue-in-cheek) Replication Platform Comparison.

Learn more about Oracle Golden Gate in the Enterprise!

Don’t settle for the same technology every time. Deploy Outside the Box.

The post Replication (Ace Comic) appeared first on Steve Karam :: The Oracle Alchemist.

We’ll call it the NoSQL Pessimism Variable: The number of search results you see before you find a page disparaging NoSQL as a viable datastore option. These pages are usually either good natured posts on why it’s a neat idea but ultimately not viable, or posts straight out slamming the platform. So too will you find the opposite; posts saying that RDBMS is the ancient past and NoSQL is the way of the future. Going by these articles you would think that RDBMS were a scourge led by Lord Sauron himself against the free peoples of Data Earth, and could only be stopped by the Fellowship of NoSQL and its distributed MongoDB Hobbit cluster.

And then of course there is the supposed animosity that exists between developers and DBAs, with DBAs thinking all developers are out to get them and developers thinking DBAs can be circumvented (but not always).

As you can probably tell from this Database Diversity series, I’m a huge fan of growth through learning about new technologies and lobbying for a place at the table. And there are of course many other forward thinkers such as Cary Milsap with his excellent article NoSQL and Oracle, Sex and Marriage (some concepts of which are included in this post). So where does the animosity come from? What is the problem some people have with NoSQL and does it have a place in our database inventory?

The ACID Test

Many articles that reference NoSQL as a poor substitute for RDBMS will bring up ACID compliance, and judging by Google there’s a lot of questions around it. So let’s take a look at ACID for a moment:

• Atomicity – A transaction as a whole either succeeds or fails, never anything in between.
• Consistency – Data must always be valid by following application logic (keys, constraints, etc)
• Isolation – No matter how much concurrency a system may have, transactions should committed and viewed in a synchronous manner
• Durability – Committed transactions must be safe in the event of a critical event

Well sorry folks, going by those definitions I’m willing to bet 99% of the actual production databases out there, RDBMS or otherwise, aren’t ACID compliant. Unless your applications perform every transaction perfectly, every column has proper constraints and error checking at all times, parameters like commit_logging are not being misused, and you can guarantee absolutely 0 data loss (not a single committed row) in the event of a nuclear strike on your data center then you’re not ACID compliant though your database system might be capable of being so.

And as Cary Milsap points out, how often is Isolation a concern anyways? How Durable is your database environment? Are you absolutely sure every required NOT NULL or UNIQUE property is actually set properly? Of course some of these things truly are a concern to some systems and thus you need ACID. Or do you?

Just because a database is NoSQL does not mean it’s not ACID compliant, and just because a system is not ACID compliant does not mean it’s not A, I, D, CID, etc. compliant. It may be that a custom combination of those four letters is all you truly need. For instance, FoundationDB promises full ACID compliance with a NoSQL engine, and systems like MongoDB, Riak, Cassandra, MarkLogic, and others can be far more durable than all but the most expensive of Oracle database environments.

No isolation? No problem! There are many times I want transactions to batch into the system and don’t really care if they’re committed or not. Some good examples are session metadata, audit information, usage history, historical data, and staging data. Load it quick and don’t worry about isolating if it gives me better performance. For that matter, forget atomicity too since I can just reload it if need be. While we’re at it, forget consistency. Some data just needs to be dumped, rules be damned.

In fact, I’d like to propose a new acronym for litmus testing NoSQL Engines: LSD:

• Limber – Flexible storage options for all types of data in multiple formats including columnar, document, key:value, or others
• Schema-less – Data need not be standardized or conformant to a strict design or datatype rules
• Distributed – Capable of spreading work over a large quantity of systems for elastic scaling when it really matters

I was going to use ALKALINE but it was just too long and K is annoying to start words with.

NoSQL at the Big Kids’ Table

NoSQL has been around for a long time, and it deserves a trusted spot in your enterprise architecture if the tool fits the need. No, you shouldn’t just migrate all of your production relational databases over to MongoDB because you read somewhere that leveraging it produced a significant ROI. But if the shoe fits, it’s your shoe. Use it when it makes sense.

Many DBAs I speak to fail to even see the reasoning behind it or why it is so popular with the development community. The pessimistic view is that developers enjoy it because they can deploy it in secret, utilizing the way of the Ninja to stealth drop it onto the server, hook their code into it, and do all sorts of conspiratorial things like efficiently query large batches of data without a single join and those pesky DBAs asking to do backups or other do-gooder stuff.

I however prefer to take a more optimistic view and say that the reason it’s so appealing is because of what it can do.

Big, Wide Rows

Have you ever had a table where a lot of descriptive details were needed in unspecified amounts, and not always guaranteed to be there? The conversation generally goes something like this:

Developer: “Alright, we need a table that can store any and all properties of items we’re selling on our website. There could be thousands or maybe none. And they’re different depending on what product it is.”

DBA: “Happy to help. Unfortunately we can’t just create a big wide table with thousands of columns. I’d suggest a table with PROPERTY_NAME and VALUE.”

Developer:

DBA: “It could get to be millions of rows.”

Developer:

DBA: “You should probably use PROPERTY_ID and VALUE_ID and join them to the PROPERTY and VALUE tables too.”

Developer:

Or something like that. In cases like this, it’s fairly clear where something like Cassandra seems appealing with its ability to store insanely wide rows and use data in the column name and query a slice range of those columns. Storing, querying, and joining tens or hundreds of properties for a single logical object as rows simply adds a ton of work. For instance, imagine a record describing a car sitting in a CAR table. You might have things like MAKE, MODEL, etc. to give an overall description. Now imagine you have a CAR_PROPERTIES table with every possible property a seller might want to provide: color, horsepower, intake, exhaust, seats, seat material, etc. Now I want you to write me a simple high performance query that finds all Hondas that are blue (if color is defined), with leather seats (if defined), racing stripes (if defined), and a moonroof, sunroof, or convertible (if defined).

Then there is the whole question of relational parent-child storage to begin with instead of storing logical groupings as a single document. For instance, consider a web based contact management system which must include businesses, people, contact methods, locations, and extra properties all on a single page. You’re not worried about atomicity or isolation at all, it’s just a dumping ground for contact information.

With a traditional relational database, many tables will have to be joined together to get all the data back for that page. If you want to make the page more reactive and allow asynchronous loading of various elements, you will have to program additional queries in to read/change parts of the page. If you want to build a search page that contains multiple contacts and all their associated metadata you’re in even more trouble. But with document storage offered by many NoSQL options, you can go from this:

To this:

A document = A page. As simple as that. Instead of crawling through nested loops and bringing data together from a multitude of locations, logically storing a chunk of application data in its native format is appealing. Is it always the right way to do things? Of course not. But it’s extremely intuitive, easy to develop against, and can be configured for concurrency, portability, redundancy, and nearly anything else you might want depending on the NoSQL engine you’ve selected.

Conclusion

NoSQL is not THE answer, but it is absolutely AN answer. In fact, I am a strong advocate of using it together with applications that use or should use a relational model. For transactional use requiring ACID compliance (most notably atomic, sequential operations) data can be stored in the relational database. Metadata, session data, logging, metrics, and other non-transactional information that is not required for the actual end user experience can be easily stored in one of many forms offered by the various NoSQL engines.

The post Putting NoSQL In Its Place appeared first on Steve Karam :: The Oracle Alchemist.

April 9th, 2013 is Internet of Things Day!

Happy Internet of Things Day! #IoT

— Kevin Ashton (@Kevin_Ashton) April 9, 2013

The futurist in me wishes that meant it is the day we all sift through our Drawer of Things to find our favorite sensors and collect all the extra data we can to share with the world, but it is not yet to be.

For now, check out my articles on the Internet of Things and how it can and will make a difference in our lives!

• Sensors Everywhere in the Internet of Things

• Wearable Computing and Personal Data Collection

The post Happy Internet of Things Day! appeared first on Steve Karam :: The Oracle Alchemist.

Since today is Monday and many of my devoted readers are at IOUG Collaborate 2013 (C13DEN) I decided to post one of my more lighthearted stories. It’s a harrowing tale of how I was a victim of ninja trickery — no, let’s not mince words, it was a strike — at my local Wegmans. If you’re only here for the technical stuff (coming soon to a blog near you) I apologize, but some stories must be told.

Ninja strike at Wegmans

Local shopper baffled – September 30, 2012

Today I was the victim of a ninja attack. If you’re one of those naysayers who don’t believe the ninja walk among us but conceal themselves to shady bars in the darker parts of Tokyo, you may go but know you have been warned. If you understand the severity of the ninja threat, read on.

So I was shopping at Wegmans, which I now regard as a hotbed for ninja activity. It is a chaotic large grocery store with many aisles and stands which are perfect for cover, and metal beams crisscrossing the roof which everyone knows will be used for transport of ninja soldiers. I had come with a list and was down to my last two items. The cart was relatively full, stocked with a variety of unique goodies that could not be easily mistaken for any other cart. I went down the ice cream aisle, intent on finding Rice Dream Ice Cream for my daughter.

Here’s the setup: I leave my cart at the end of the aisle, and walk down toward where the organic ice creams are. There is a man across from me, nondescript, blue jeans, a faded t-shirt, no one that you would remember. As I turned to the frosted doors to begin my search, I heard the dull slap of a dropped box of ice cream, and turned to look. The man stood there, looking down at a bland box of generic ice cream pops. So generic, in fact, that the makers of them didn’t even have the heart to call them ice cream pops; instead they were “Frozen Treat Pop Product” or something like that. The man didn’t look toward me, he didn’t bend to pick them up in shame, he just looked at the frozen product on the ground. Losing interest, I continued looking for my daughter’s ice cream, couldn’t find it, and walked toward my cart.

The cart was not there. I was the victim of cart theft. Petty cart theft, but cart theft nonetheless. It’s something you hear about but never really believe…who would steal a cart?

But steal it they did. I looked around the store for a good ten minutes, going up and down aisles, giving every cart I passed the hairy eyeball, glaring down old women and couples and little kids. But my cart was nowhere to be found. Finally I decided my only action could be to start my shopping excursion all over and get a new cart. So I did.

I took my empty cart to the first item on the list once again, which were apples. I looked down at the apples and picked a few which were okay but obviously not as good as the glorious red delicious fruit I had picked out previously. Finding a few to bag up I turned… and my cart was sitting right next to me. The old cart. With everything in it…plus the generic box of frozen ice cream pops sitting in the top basket. I considered the basket with new suspicion, and realized that something was very wrong, something far more sinister than a simple petty cart theft. But what happened then was even more disturbing. I put the frozen ice pops in the empty cart, triumphantly turned my old cart around to get my final items, turned…and the cart with the ice pops was gone. Vanished. Nowhere to be seen.

Ninja. The word reverberated in my head. I felt a little dizzy. I was not ready for this.

Getting my final items I went to the check out counter. The lady asked if I found everything okay and I told her my tale of weirdness. She laughed and said that people accidentally take carts. That poor naive woman. She had no idea. She finished checking me out, bagged my items, and I walked to the back of the cart to finish this strangeness once and for all. Turning the cart to the right to leave the check out lane, I found it blocked.

Blocked by a cart with nothing in it but a box of Frozen Ice Pops.

True story.

The post Victim of a Ninja Strike appeared first on Steve Karam :: The Oracle Alchemist.