DROP DATABASE command
I have been DBA-ing for a while now, and I today I used a “new” command which I have never used in the previous 20+ years I have worked with Oracle: DROP DATABASE. It’s amazing what you miss sometimes!
So, what does it do? As the name implies, it drops the database. That is, it deletes all of your database files. This is significantly safer than using traditional Operating System methods (assuming you even have access and are not using ASM), and for added safety you have to be running the database in RESTRICT MOUNT mode.
What a lovely command. It goes to all the trouble of locating and removing those troublesome files that were put into the wrong place by another DBA that you would have missed if cleaning up manually.
It certainly made the pre-production clone script (cloning from the Physical Standby) that I have just written quite a bit easier to code.
Does anybody know if there is any other reason for running the database in RESTRICT MOUNT mode, or was it designed just for this command?
EDIT: I was just informed via Twitter (@chandlerdba) by @pfierens that this also takes out your SPFILE. If you are using an SPFILE (you should be!) and want to keep your db parameters for a db rebuild, I would recommend creating a pfile from the spfile first…
Has anyone ever done an RMAN “drop database including backups” ? You REALLY need to be sure you don’t want it back to run that one!
SAN Migration: When modern SANs “Fail”
… or at least, when they don’t behave like you were expecting them to.
I recently performed a major migration of a dozen missions critical Oracle and SQL Server systems from a pair of old EMC CX700′s to 2 brand new shiny HP XP 20000′s. This blog post is intended to show, at a very high level, what steps were taken to ensure the new SAN could cope with the volume of traffic. It also shows that, despite the new SAN being much more modern and intelligent, how you can still get caught out by unexpected behaviour.
The databases to be migrated varied from high throughput OLTP, low throughput – fast response time OLTP, a small Data Warehouse and Grid Control. The systems are 24 x 7 x 365, but a nightly maintenance window can be arranged on some days of the week between midnight and 3am. Not a large window, but invaluable.
What are we migrating?
For a successful SAN migration, we first need to understand what we are migrating. To do this we need to profile the I/O requirement of each Oracle database. Oracle provides us with all of the tools we need to be able to do this. If you have the Tuning and Diagnostics Pack licensed (which all Oracle sites should have – it should really be part of base product.) then you can pull the information you need directly out of the AWR table DBA_HIST_SYSTEM_EVENT. By default your AWR stats gather every hour and only last a week. Hourly averages of system event wait times will give a good starting point, but I would not want to use them to determine peak volumes. If you capacity plan to your averages, you will run out of resources before you hit your peaks. A lot can happen in an hour when you are looking at millisecond response times. You need to be much more granular than an hourly average, but you might not want change AWR to gather all of its information as frequently as you need for the I/O events. You need to develop some code to gather the information from v$system_event directly, so I did [I will post this code up shortly]. You can be as granular as you want, but I balanced peaks against volume of information and the impact of gathering it and was running every 5 minutes. Mostly. I also took some much more frequent samples to ensure that I wasn’t wildly wrong with the 5 minute averages, and that I wasn’t experiencing very short peaky loads. This method also has the advantage of not needing any additional pack licensing from Oracle. Sorry Larry.
What are the resources available on the new SAN?
The second requirement for a SAN migration is to understand the capabilities of your new SAN. There were already some systems on the new SAN so I was unable to test to destruction. However, some dedicated resources were available on the shared SAN, namely dedicated disks, cache and front-end ports. I needed to throw substantial workloads at the SAN to stress the resources and find the tipping point. There are many workload tools available, such as Hammerora and RAT. Some are expensive, such as RAT. However, for my purposes I was only looking to stress the SAN and get a comfortable feeling about the IOPS and throughput performance. This was achieved using Oracle’s free SAN stress tool, Orion [I will hopefully post more details about this shortly]. By taking all of the profiled I/O rates, adding them up and comparing against the new SAN and it’s much greater cache capacity (16GB dedicated to my systems, up from 4GB on the EMC’s), we were able to see that the footprint of the databases to be migrated fitted comfortably into the capabilities of the new SAN. And so the migrations began.
Thin Provisioning
To provide resilience for the databases, LUNs were presented from each SAN to each cluster node, and were mirrored at the Operating System level, either using native mirroring on the Unix platforms, or Veritas on the Windows platforms. This allows either SAN (or SAN location) to fail with no loss of service. The LUNs were either RAID-10 or RAID-5, and were carved out of the storage pools using Thin Provisioning.
Thin Provisioning (or using Meta-Luns or similar techniques from your SAN provider) is a way to allow LUN’s to be easily carved out of storage pools and extended at a later date. They are wonderful for SAN administrators and people who have not capacity planned their systems due to their dynamic nature and minimization of space wastage. However, they work on the basis of providing a bit of storage to your LUN from every disk in the array. This gives a maximum concurrency, maximum contention model. Every LUN is on the same disk competing for the same resource. Lots of IOPS, but everyone wants them. The Thin Provisioning also adds a small overhead to the I/O processing. Not much. Not even a whole milli-second.
Migration Problem
The migrations progressed well, from least critical to most critical system. As each system was migrated, we kept monitoring and measuring the I/O response times. Migrating the Data Warehouse showed that the new SAN was performing as expected. Processing times were lower. Not much, but in line with expectation and allocated resources. However, Grid Control was exhibiting some strange response times. Despite theoretically having more resources, some I/O response time metrics were worse on average, and much more variable than before. With the EMC SAN’s, we had experienced a good, consistent level of performance for each LUN. The XP’s were proving to be more erratic.
I did a lot of investigation, moving resources around (e.g. dedicated front-end ports for the Grid Control LUN’s.) and measuring the affect. The level of I/O associated with Grid Control, with only a couple of hundred servers under management, was substantially lower than that of systems competing for shared resources (e.g. the Data Warehouse, which was exponentially more demanding). It seemed that Grid Control’s LUNs were not able to survive in the SAN cache. The old EMC cx700′s were relatively “dumb” compared to the new XP20000′s, and had effectively dedicated an amount of cache to each LUN. The more intelligent XP’s would not be so potentially wasteful of resources, and so deemed the (relatively) low level of activity from the Grid Controls LUN’s to be unimportant and aged the blocks out.
We could live with this for Grid Control. Blisteringly fast response time is not a critical factor in the success of Grid Control. The same could not be said for one of our mission critical applications. It requires a very low end-to-end response time for each transaction. We carefully measure each step – client, calculation, render, network latency, database response time, etc. It needs to be fast and consistent. From the database, it really needs to hit cache or SSD’s. Spinning rust (ordinary disk) is simply too slow. However, the level of activity for this system, in terms of MB/s and IOPS, is relatively low compared to all of the other systems competing for SAN resources. About 3% of the whole. Add to this the overhead of Thin Provisioning and we have a problem. When a 2ms average turns into a nearly 4ms average with much higher peaks, as it did for db_file_sequential_read on Grid Control, that would be a major problem for this application.
Solution
Talking to HP, they would only guarantee the response time we demand by using SSD’s and a cost that made my eyes spin and think of buying a Supercar instead. A more practical (i.e. cheaper) solution had to be found, and it was (with the help amd support of HP, just no guarantees 
). We bought some more conventional disk and configured it into a traditional Raid-10 storage pool. No Thin Provisioning. We then partitioned the 16GB SAN cache into a 4GB cache dedicated to this system, and left the 12GB cache behind for everything else. We were migrating from 4GB, and understood that we had plenty of wriggle room in this configuration.
And the result? After migration, the performance of the mission critical low throughput system improved substantially, with consistent log_file_parallel_write times of less than 2ms and db_file_sequential_read times of less than 1ms, better than we were experiencing on the old reliable EMC’s. I mean, less than 1ms? That’s fast.
I/O Response Times – Graph
UKOUG 2011
Well, I’m back at work today missing the excellent final day of the UKOUG 2011 (@UKOUG #UKOUG2011), but frankly I’m worn out.
After watching some of the most fantastic presentations by the likes of Doug Burns, Jonathan Lewis, Greg Rahn, Tanel Poder and many others**, my brain is full. I couldn’t absorb another fact.
A similar crowd, too numerous to mention, then helped me destroy the information that I had just learned, through the medium of good British Real Ale and Whisky.
Technical blogs to follow, once my brain cools down and my liver recovers.
Neil.
**A special mention to Thomas Presslie for pouring shots of The Balvenie Scottish Single Malt Whisky into me to demonstrate DataGuard Fast Start Failover, at 10:30am. Transactional Whisky indeed. Not sensible but good fun and a novel way to represent a commit 
– I’m on the left http://lockerz.com/s/162506859 (thanks for the Photo, Doug)
Industry Experience
I don’t get it. Why do so many jobs and contracts seem to insist upon having experience in a particular industry when, in the overwhelming majority of cases, the specific industry in which we work has no bearing upon the nature of our work.
I have worked across many industries, but each time I talk to a recruitment agent I get similar questions: “Have you worked in X industry?”, “I won’t put you forward for Y unless you have worked for Z”.
It’s the wrong question. Have I worked in Media? Investment Banking? Accountancy? Property? Logistics? It doesn’t matter. No, really. It doesn’t. I have worked in all of those industries and a few more besides, and the nature of the industry was largely irrelevant. A friend recently suggested that you need Investment Banking experience so you understand the inordinate bureaucracy and dreadful boredom that come with working for an Investment Bank. A little unkind, but I know where he’s coming from.
What is relevant is the type and nature of systems with which you are working. Are they mission critical? Zero downtime? Very High Transaction rate? Enormous Data Warehouses? Hundreds or Thousands of databases? These questions have relevance. A high transaction rate OLTP in a Bank is very similar to a high transaction rate OLTP Web Retailer. The challenge with these systems is a different to that of an enormous data warehouse, but it’s still fundamentally an RDBMS. Data is data is data. We don’t need industry experience – it doesn’t help us in the same way as it helps Business Analysts or Project Managers or even Developers.
The recruitment problem for DBA’s is that recruiters don’t know the difference between OLTP and Data Warehousing; a large proportion simply keyword match (the great ones don’t! – and there are genuinely great recruiters out there, in small numbers) so you need to ensure you have all of the relevant keywords on your CV – I have even been asked to amend my CV to put Word and Excel on there! WTF? Unfortunately you also need to be careful, otherwise you’re probably getting job adverts sent through for Cobol Programmers, Websphere Guru’s and all manner of support and helpdesk staff. I removed IBM Assembler Programmer from my CV about 10 years ago, although I suspect there are not too many jobs left for that skill set now.
Implicit Conversion Errors
A while ago, I failed over a database (as planned) to it’s Dataguard copy, and of course everything worked as expected. Everything, that is, except a couple of reports which get sent directly from the database server early in the morning. The report generation had failed. After some investigation, we discovered that the newly active Dataguard server did not have NLS_DATE_FORMAT set in the environment, and the 2 reports in question were coded something like:
select col1, col2, col3 from user_data where user_date >= '25-Nov-2011 00:00:00';The select was failing with the error. ORA-01821: date format not recognized
If the developer had coded an explicit conversion, then we would not have experienced a problem.
select col1, col2, col3 from user_data
where user_date >= to_date('25-Nov-2011 00:00:00','DD-MON-YYYY HH24:MI:SS');
Coding with an implicit date mask is great and works successfully every time, as long as the NLS_DATE_FORMAT in your session matches the date mask you have supplied, which is course it always does. Until something changes and it doesn’t. In my experience, implicit conversion is probably the single greatest source of failure in systems and also one of the hardest to track down. It frequently occurs in a badly designed schema which doesn’t use the correct datatypes. I have seen schemas where everything is being stored a VARCHAR2, including numeric fields. This works fine as Oracle will happily insert implicit to_number functions into your code and return answers in ways which seem correct, until you get some rogue data into the database and everything falls apart.
USER1 @ orcl > -- Create a table but allow generic data, rather than specifying numeric data USER1 @ orcl > -- The client will take care of validation. Of course it will. USER1 @ orcl > create table implicit_problem (col1 varchar2(10), col2 varchar2(10)); Table created. USER1 @ orcl > USER1 @ orcl > -- Lets fill the table with reasonable data USER1 @ orcl > insert into implicit_problem values (1,1); 1 row created. USER1 @ orcl > insert into implicit_problem values (2,10); 1 row created. USER1 @ orcl > insert into implicit_problem values (3,66); 1 row created. USER1 @ orcl > USER1 @ orcl > -- Oracle is putting an implicit to_number around col1*col2 to allow the calculation USER1 @ orcl > select col1,col2,col1*col2 from implicit_problem; COL1 COL2 COL1*COL2 ---------- ---------- ---------- 1 1 1 2 10 20 3 66 198 USER1 @ orcl > USER1 @ orcl > -- And now lets have some incorrectly validated data USER1 @ orcl > insert into implicit_problem values (4,'A'); 1 row created. USER1 @ orcl > USER1 @ orcl > USER1 @ orcl > -- And now the implicit conversion fails USER1 @ orcl > select col1,col2,col1*col2 from implicit_problem; ERROR: ORA-01722: invalid number no rows selected USER1 @ orcl > USER1 @ orcl > -- Cleanup USER1 @ orcl > drop table implicit_problem; Table dropped.
It’s much easier (and quicker) to catch bad data going into a system than it is to perform problem resolution. Always code explicitly for your data types. Implicit conversion in yuor coding invariably leads to hard-to-find bugs.
The 10046 trace. Largely useless, isn’t it?
The other night I was sat in the pub with some like-minded individuals discussing the relative merits of the 10046 trace (we Rock! in the pub, dudes!) and somebody asked me how often I has actually used it in anger? A well-respected DBA / Architect maintained it was a pretty useless and difficult option to use, given the topology of modern applications (e.g. How do you find the correct session with all that connection pooling going on from multiple web servers.)
My answer surprised me – I thought back to one client where I spent 90% of my time performance tuning a large (TiB-ish) OLTP/Batch hybrid system and concluded that I had ran a 10046 against production about once a year. Once. So if the 10046 is the holy grail of plan information, why wasn’t I using it that much. And why did I never use a 10053 against Production there?
The answer for me is a little more complex than that given in the pub:
1. as stated above, it’s hard to catch the in-flight session unless the application is instrumented to inject the trace statement when needed (and how many applications are instrumented to help you discover problems? Screen ST03 in SAP is very helpful. Any others in major ERP’s? Thought not.)
2. In many places that I have worked, getting authorisation to make any a change to a 24×7 mission-critical system is highly bureaucratic, involving cast-iron justification for the change and it’s positive benefits, requirement that there will be no adverse effects because of the change, very senior sign-off, more red-tape, etc. This causes a significant amount of work simply to put a trace on, even if you can catch the SQL. This can end up being more work than actually fixing the problem.
3. An awful lot of SQL tuning is a fairly blunt affair, as the developer (who is frequently database-blind) has usually missed something obvious. It is frequently to do with incorrectly using or not using an index (or using a poor index), or lack of filtering data at the right point to minimise the I/O.
4. Most importantly, if you have AWR and ASH, it’s not really needed. For each plan created by the optimizer the database stores the bind variables along with it, so we can usually understand why the optimizer makes the decisions it makes. ASH contains the main event waits. Why bother trying to capture all of the detail in a trace when you really don’t need that much detail, and it’s all already there; ready to be extracted from the relevant tables (e.g. dba_hist_active_sess_history, dba_hist_sql_plan and dba_hist_sql_bind.)
I have never used a 10053 trace on a Production system. I have simply never needed to know the decisions taken by the optimizer in that much detail. Like most DBA’s and Oracle consultants, I don’t go from site-to-site on a weekly basis resolving edge-case problems that the incumbent DBA’s haven’t had the time, or possibly don’t have the skills, to resolve themselves. I usually don’t need that level of confirmation that I’m right about why the plan is wrong, and I don’t have the time to conclusively prove it over and over again – I just need to get the fix into place and move onto the next problem.
That said, perhaps I should get fully to the bottom of these problems to ensure that they never occur again – which is the fundamental problem with Adaptive Cursor Sharing.
Oracle Timestamp Processing – mildly annoying
I was writing a small piece of SQL this morning which I needed to account for daylight savings time correctly. All of my databases run in UTC, so a quick foray into using TIMESTAMP AS TIME ZONE seemed the easiest way to accomplish this. So, I code it up and want to test my code to ensure that the timestamp operates correctly on both sides of UK Daylight savings. I figured that the easiest way to do this would be to used the old Oracle initialisation parameter FIXED_DATE. You can set this in the database on the fly and observe the results immediately. So to for testing (in a Dev database which only I was using). Guess what? FIXED_DATE works perfectly for SYSDATE. However, it is completely ignored for SYSTIMESTAMP! WHAT??? Who in Oracle missed this one? Let me show you how this (doesn’t) work, with a workaround for my testing included in the example to show how neatly TIMESTAMP AS TIME ZONE does work:
> alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS' Session altered. > select * from v$timezone_names where tzname like '%London%' TZNAME TZABBREV ---------------------------------------------------------------- ---------------------------------------------------------------- Europe/London LMT Europe/London GMT Europe/London BST Europe/London BDST Before we start, check the current date > select sysdate from dual SYSDATE -------------------- 01-NOV-2011 11:50:15 Before Daylight Savings Time changes (on 29th October 02:00:00) > alter system set fixed_date = '28-OCT-2011 08:00:00' scope=memory System altered. > select systimestamp at time zone 'Europe/London' from dual SYSTIMESTAMPATTIMEZONE'EUROPE/LONDON' --------------------------------------------------------------------------- 01-NOV-11 11.50.15.666000 EUROPE/LONDON The timestamp is unaffected by setting FIXED_DATE!!! Oracle! Grrrr! > select sysdate from dual SYSDATE -------------------- 28-OCT-2011 08:00:00 The UTC time is correct. So I need to take the sysdate and transform it into a timestamp at timezone London should be 1 hour ahead of UTC at this point > select to_timestamp(sysdate) at time zone 'Europe/London' from dual TO_TIMESTAMP(SYSDATE)ATTIMEZONE'EUROPE/LONDON' --------------------------------------------------------------------------- 28-OCT-11 09.00.00 EUROPE/LONDON ########################################################################### Now to roll the time on and re-test ########################################################################### After Daylight Savings Time changes (on 29th October 02:00:00) > alter system set fixed_date = '30-OCT-2011 08:00:00' scope=memory System altered. > select sysdate from dual SYSDATE -------------------- 30-OCT-2011 08:00:00 Now Daylight savings should by the same as UTC, not 1 hour ahead > select to_timestamp(sysdate) at time zone 'Europe/London' from dual TO_TIMESTAMP(SYSDATE)ATTIMEZONE'EUROPE/LONDON' --------------------------------------------------------------------------- 30-OCT-11 08.00.00 EUROPE/LONDON Yey! Daylight davings is correct for London. Remove the fixed_date setting > alter system set fixed_date=NONE scope=memory System altered. And check the date > select sysdate from dual SYSDATE -------------------- 01-NOV-2011 11:50:16 ------------------------------------------------------------
There you have it. How very mildly annoying. Can’t use TIME ZONE with SYSDATE, can’t use FIXED_DATE with SYSTIMESTAMP.
Dennis Ritchie RIP
There can be few scientists who have contributed so much to the world as Dennis Ritchie. Completely anonymous to the world at large, and to far too much of the computing fraternity too, his involvement in the development of C – the first portable programming language, and Unix cannot be understated. Who uses Unix? Well, everybody. What is the O/S on embedded devices? What did Steve Jobs base his Mac O/S and Mobile Operating systems on? Upon what did Bill Gates base the operating system for the new IBM PC in 1980? What do corporations across the world use to power their servers? What is it all built on? Unless you’re running some proprietary mainframe software, the building blocks of what you are using were set, to some degree, by Dennis Ritchie.
Whilst I never met him myself I did work with his sister, Lynn, for several years at a software house in the North East of England, where she still resides. She told me some of the stories about him; how the invention of C and Unix were really aims in making computing more portable, more standard, easier to use. How all that Ritchie, Ken Thompson, Brian Kernighan and the others got for this was their salary [although they did make a few quid on top out of their books ], and the ability to distribute Unix free, with C, to all and sundry as Bell labs, a subsidiary of AT&T, had no financial interest in computing as they were a regulated telephone monopoly at the time.
Thanks Dennis.
Management and Infrastructure SIG – Thank You
Well, we had the final Management and Infrastructure SIG last week, and whilst the attendance wasn’t huge, the content was simply excellent. Two great presentations about Enterprise Manager from Niall Litchfield of Maxima, and from Mark Westwood and Carl Holmes of Morrissons.
We also had an insightful and revealing presentation on outsourcing from Piet De Visser from Logica, and an eye-opening technical feast showing how to maximise the performance from your Intel hardware from Steve Shaw of Intel.
I would also like to thank Oracle for their support, and especially Mike Edgington for his talk about the Oracle support organization and how it is evolving.
Ye Gods, this feels like an Oscars acceptance speech!
Many thanks to the UKOUG Staff in general, and specifically Marisa Harris and Jake Potter for their organisation and assistance in making this SIG work.
Finally, I would like to thank Martin Widlake for his Chairmanship, and his opposition during our presentation about what DBA’s really need to know.
And so ends the UKOUG Management and Infrastructure SIG, now to be merged with the RAC SIG and be reborn as the UKOUG Availability, Infrastructure and Management (AIM) SIG, where we will still talk about everything that we have previously, but extend the remit around RAC, Exadata, Goldengate, Partitioning, VLDB’s, and all those other lovely cost-extra options that Oracle dearly would like to sell to us all.
Thank you to everyone who attended the M&I SIG – we hope you got as much out of it as we did.
