The Internet doesn’t forget either.  Recommendations that were once helpful, if not outright essential, now are neutral or even downright bad. So now, whenever I put together new presentations, I always spend a lot of time in research, reassessing my knowledge on the topic.  (That doesn’t mean that I’ll extemporaneously say it wrong, because I speak in an off-the-cuff and rapid style. But at least my notes are usually correct).

Case In Point

Take backup and recovery (B&R), for example.  I’ve been writing about and performing B&R for years.  I’m even part of the team that builds the most popular B&R tool in the SQL Server space.  It’s an extremely important part of what I do.  And yet, even after spending a couple days re-researching topics (yet again), I still manage to get a few of the more specific details wrong because of changes over the years in the feature set.  How so?

During an Expert’s Perspective webcast last week (and available on-demand) covering the top backup and recovery mistakes on Microsoft SQL Server, I mis-explained the details concerning how SQL Server performs a differential backup.  A very clever SC on our team who was also attending the webcast pointed out to me after the session that I’d described differentials working at an 8k page-level when, in fact, they work at the extend-level (that’s a block of eight 8k pages). #FACEPALM!

When describing the differential backup I said that a bit was flipped on each page header and SQL Server would only back up those pages. Instead, whenever a page is changed a bit is recorded on the differential change map (1bit per extent), the backup process then queries these map pages and backs up those extents that have been marked as changed.

• Watch the on-demand video of Performance Tuning SQL Server for Backups and Restores, with me, Iain Kick, and Brent Ozar (blog | twitter). Free, but registration required.
• Watch the on-demand video of Ten Incredibly Painful SQL Server Backup and Recovery Mistakes, with me and David Gugick.

The Research Reveals

The cool revelation here is not that I can admit my mistakes.  The take-away from this blog post are all the great articles I read writing my latest slide deck.  These blogs and articles were so good that I needed to spread the word.  Add these to your must read list:

Virtual Log Files must be tamed!

• Performance impact: a large number of virtual log files Part I and Part II by one of my favorite not-so-prolific bloggers, Linchi Shea (blog). The thing I love about Linchi’s content is its rich benchmarking information.  Few SQL Server experts out there routinely test the behavior and performance of specific SQL Server features like Linchi does.
• Slow recovery times and slow performance due to large numbers of Virtual Log Files, by Graham Kent (blog) of Microsoft Sweden, shows that VLFs impact not only the processing time of transactions but also B&R processes.
• Lots of VLFs are Bad, by Tony Rogerson (blog | twitter), was the article that first got me interested in this behavior.

Backup and Recovery Myth Busting

• Breaking the Backup Chain – Redux (Or ‘Eating Crow’), by Michael K. Campbell (blog), shows that out-of-band database backups break differentials not transaction log backups.
• Backup with COPY_ONLY, How to Avoid Breaking the Backup Chain, by the eminent Paul Randal (blog | twitter), uses clear writing and first-class demos to illustrate the principle of breaking backup chains in Microsoft SQL Server.  Other great blogs of a related nature from Paul include:
• Debunking a Couple Myths around Full Backup
• The Importance of Validating Backups
• 30 Backup Myths

Other Good Sources

And just in case you need a refreshing in the basics of SQL Server backup and recovery, check these out:

• Backup best practices from Brent Ozar
• Backup 101 – recovery models and transaction log management

I know that I did just that, for years.  It wasn’t until I was trying to performance tune some existing code that I realized I hadn’t actually taken caching of data and execution plans into account.  So all those modified stored procedures that I was so proud of might not actually be faster than the first generation of procedures because I hadn’t checked to ensure that I was testing cached programs against uncached programs (and, by extension, the data used by those programs).  That’s easy enough to fix with a test harness.  Test harness were originally an actual, physical harness used by engineers to clamp down parts of an electrical or mechanical device they were prototyping.  Ours is no different.  It locks down all of the assumptions about our code (like my early, false assumption that I didn’t need to clear the caches) and adds a metric or two for good measure – literally – so we can better measure what’s happening in that code.

Here’s what my test harness looks like:

/* Transact-SQL test harness by Kevin Kline, http://KevinEKline.com, Twitter at kekline */

/* Flush dirty pages from the buffer to the database files. */

CHECKPOINT;

/* Flush the data cache and procedure cache, respectively. For DEV environments only! */

DBCC DROPCLEANBUFFERS;

DBCC FREEPROCCACHE;

/* Enable statistics tracking for IO and timings. Remember, SET commands remain enabled during a session until disabled. */

SET STATISTICS IO ON;

SET STATISTICS TIME ON;

-- Whatever SQL code you'd like to process goes below.

SELECT SalesOrderID

FROM Sales.SalesOrderHeader H

WHERE CustomerID = 344

GO

SET STATISTICS IO OFF;

SET STATISTICS TIME OFF;

/* Textual Execution Plans, if desired.

SET SHOWPLAN_TEXT ON;

SET SHOWPLAN_TEXT OFF;

*/

I also like to include the execution plans a lot of the time.  You might wonder why I don’t save the execution plans for the GUI in SSMS?  Well, I’m a big advocate of scripting in general because I like to automate activities.  By pulling the execution plans using scripts, I can use SQLCMD to schedule a large number of query executions during the evening and have the results ready for analysis when I come back into the office in the morning.  Workin’ smarter, not harder, Baby!

So how does this test harness work for you?  Do you use other elements in yours?  If so, share your experiences here!

Thanks,

-Kevin

-Follow me on Twitter

“Login failed for user ‘<user_name>’. (Microsoft SQL Server, Error: 18456)”.

The challenge when troubleshooting this error message is that you may have a problem with SQL Server or you may have a problem with Active Directory or Kerberos, if you’re using one of those authentication technologies.

A False Trail

When times are good, and you’re able to make a connection, you can always query sys.dm_exec_connections, to see how you’re connecting, for example, using NTLM rather than Kerberos.  But that doesn’t help us when we can’t connect to the server at all.

In other situations, your problem might be caused by duplicate SPMs in Active Directory.  MVP Russell Fields documented a nice solution for ridding Active Directory of duplicate SPNs here.  Microsoft Support also mentions some troubleshooting steps forauthentication problems here.  Ok, that helps. But it’s not my solution. 

MVPs to the Rescue

Fortunately, my MVP buddies Edwin Sarmiento (blog | twitter) of Canada and Bitemo Erik Gergely (blog) of Hungaria had already discussed and solved the problem for me!

Something Erik pointed out, but hadn’t occurred to me at first, is that if you’re getting this SQL Server error message then you’ve actually reached the server and probably aren’t having a full disconnect error.  The second thing that Erik pointed out is the importance of the state element of this error message.  A lot of the time, you can simply ignore the
state element of an error message.  But not this time.  As it turns out, state is the key to solving the problem.  For example, a state of 18 indicates that the password must be changed.

Microsoft provides a pretty good description of the states of error 18456 here, but it leaves out a few things.  (You’ll get more useful info if you read all of the comments too).  But
again, Erik comes to the rescue by providing a complete and concise list of error 18456 states here.

Microsoft Improves the Documentation

Even better for all troubleshooting situation involving state information, Microsoft has now added state descriptions for errors in Books Online, including error 18456.  When you look in your SQL Server error log, you see the state of the error and be able to make an accurate deduction about  the nature of the error!

Hope this helps,

-Kev

-Follow me on Twitter

Here’s a run-down of several useful documents and downloads to improve overall operation performance for those of you in a Microsoft-centric IT organization:

Microsoft Operations Framework

Microsoft Operations Framework (MOF) version 4.0 guide is practical guidance for IT organizations. With the release of version 4.0, MOF now reflects a single, comprehensive IT service lifecycle—it helps IT professionals connect service management principles to everyday IT tasks and activities and ensures alignment between IT and the business.

Infrastructure Planning and Design

The Infrastructure Planning and Design (IPD) guides are the next version of Windows Server System Reference Architecture. The guides in this series help clarify and streamline design processes for Microsoft infrastructure technologies, with each guide addressing a unique infrastructure technology or scenario.

Microsoft Baseline Security Analyzer 2.2 (for IT Professionals)

The Microsoft Baseline Security Analyzer provides a streamlined method to identify missing security updates and common security misconfigurations. MBSA 2.2 is a minor upgrade correct minor issues and add optional catalog support.

Security Compliance Manager

The Microsoft Security Compliance Manager provides centralized security baseline management features, a baseline portfolio, customization capabilities, and security baseline export flexibility to accelerate your organization’s ability to efficiently manage the security and compliance process for the most widely used Microsoft technologies.

If you’d rather not read my entire previous article – quick recap, expressor Studio uses “semantic types” to manage and abstract mappings between sources and targets. In essence, these types are used for describing data in terms that humans can understand—instead of describing data in terms that computers can understand. The idea of semantic abstraction is quite intriguing and it gave me an excuse to use data from data.gov to build a quick demo. You can download the complete data set I used from the following location: International Statistics.  (Note: I have this dream that I’m going to someday download all of this free statistical data sets, build a bunch of amazing and high-value analytics, and make a mint.  If, instead, YOU do all of those things, then please pay to send at least one of my seven kids to college in repayment for the inspiration.  I’m not kidding.  I have SEVEN kids. God help me).

The federal government, to their credit, has made great progress in making data available.  However, there is a big difference between accessing data and understanding data. When I first looked at one of the data files I downloaded, I figured it was going to take me years to decrypt the field names. Luckily, I did notice an Excel file with field names and descriptions. Seriously, there are single letter field names in these files where the field name “G” has a description of “Age group indicator” (Oh Wow).  See the figure below.

It’s stuff like this that reminds me why we have data quality and master data management tools.  Ok, back to expressor Studio. I quickly mapped a couple of files into expressor Studio using their “Read File” operator. It was fairly simple and easy to use. My data included files with country area information, population, and gender information by year. Once I mapped these files I quickly wanted to shed the default cryptic, nay, nonsensical names. I could have just renamed the fields when I initially mapped them into the system but that would mean I would have to manage the names in three separate locations. Bah! It made more sense to create a common semantic type and reuse it across all three files.

There are two flavors of semantic types within expressor Studio to handle your mappings, atomic types or composite types. An atomic type is simply a single field name whereas a composite type is a combination of one more atomic types. Since the data files had many common fields, I decided to create a core set of atomic types that I could then roll up into composite types based on the files I was mapping. This kept the mappings simple and easy to understand and most importantly the whole exercise took about 5 minutes. Once the types were created I simply mapped the cryptic names from the files to the business friendly names in my semantic type.  (I can’t even begin to imagine how long this would’ve taken to build using native tools, but certainly not 5 minutes).

Now I was ready to move my data. I took the data from three files and combined them into one master dataset. From there, my international statistics from Data.Gov were pumped right into my waiting SQL Server database.  Note that I could’ve used Excel or just about any other database as my target instead of SQL Server.

Now, you might be saying to yourself “That looks easy because you read all the help files first.”  Actually, no.  In fact, some of my buddies like to lovingly tell me to “RTFM” from time to time.  It’s not that it offends my masculinity to read a manual.  I just usually like to have a go first and then, if needed, go back to the manual.  In fact, all I really used was this 5-minute demo video that in noticed when I was downloading the tool.

If you’re tackling ETL and you want it fast and easy, then you might want to check out their website, www.expressor-software.com, to learn more about the expressor company and products.

Enjoy!

-Kev

P.S. Follow me on Twitter!

Introducing the Microsoft Broadcaster: A self-service technical content repository that provides you with rich content for your blogs/sites.

On behalf of Microsoft, I would like to invite you to dive into Microsoft Broadcaster.  It’s a site that unleashes a torrent of free technical content like Videos, Webcasts, Whitepapers, eLearning, and more, which you can use in your blogs or user group site to drive deeper engagement with your audience and community.

Key Features of Microsoft Broadcaster include:

• Download and embed content on your sites without sending traffic away from your site with Microsoft’s full blessing.
• Customize site by product (e.g. Windows, SharePoint, or Office), by keyword, or by form factor (e.g. webcasts, videos, podcasts)
• Set alerts for content updates.

To take advantage of the offer, visit www.microsoftbroadcaster.com.  For a more detailed information and program overview please visit http://www.microsoftbroadcaster.com/en-us/faq

Now, It’s time for me to get involved and start to participate in the program!

Enjoy!

-Kevin

P.S. Follow me on Twitter!

So I’m going to start sharing what I’m reading at least once per week, partly so that I don’t firehose too many reading links directly into your brain (where I to do it say once per month) and partly to solidify in my own mind the information that I’m reviewing. So here are a few good links for the seven days leading up to July 22, 2001:

• Microsoft and Whitehouse partnership on BigData: BigData isn’t a particularly new concept.  But I was intrigued to learn that the National Science Foundation, Microsoft, and 13 other teams were partnering on developing better BigData analytics for lots of government data from activities such as healthcare, economic development, education, transportation, and the power grid.  Cools stuff!  Plus, Microsoft has developed a new tool called Project Daytona to better harness the power of the cloud, in general, and Windows Azure, specifically.
• While we’re on the topic of Federal IT in the Cloud be sure to read this linked article from ComputerWorld.  Say what you will about our government, but putting government IT in the cloud and increasing both its transparency and availability will make a huge difference in how the Federal government will be able to service the public.  We’re talking as big a difference as corporations experienced between the “catalog on the web” experience of the 1990′s to the Web2.0 experience of today.
• If you’re the social media type, give this article a read discussing the Power of Hashtags in Social Media.
• The Register, of the UK, whose tagline is “Biting the hand that feeds IT” has a great article on a spat over database technologies between the IT sage Michael Stonebreaker and Google.  It’s a great read if for no other reason than to prove that databases are worth fighting over.
• And if you think Microsoft is still towing the relational database barge without thinking about other technologies, you need to read up on Projects Dryad and Daytona.
• Finally, I’m still getting lots of questions about when and where to limit SQL Server’s Max Degrees of Parallelism.  Be sure to read Microsoft’s Recommendations and Guidelines for ‘max degree of parallelism’ configuration option here.

And just because so many of us in IT are closet or former musicians, there’s Live Guitar Lessons with Steven Krenz, sponsored by my hometown boyz at Gibson Guitar.

Got a favorite article or tool tip? Let me know!  Enjoy,

-Kev

Follow me on Twitter.

Remote Desktop Services Component Architecture Poster

Grab your own poster! A visual guide to key Remote Desktop Services technologies in Windows Server 2008R2

Virtually Free

Get the latest update rollup package for the Hyper-V role in Windows Server 2008 R2 and be sure to bookmark the Windows Virtualization Team Blog.  Be sure to check out blog World Simplified is a Virtual World. And doncha evah neglect application virtualization, such as the goodness at the App-V Product Team Blog.

Let’s Optimize Some Desktops (Assuming You Have Gone Full Cloud Yet)

Check out The Official Microsoft Desktop Optimization Pack (MDOP) Blog where you can get cool tools like the Diagnostics and Recovery Toolset (DART), currently in its v7 beta release.

We Can Always Perform a Little Better

Have you checked out the Ask the Performance Team blog yet.  You should!  Then there’s the Microsoft Enterprise Networking Team blog.  And if you’re looking for help with the Windows Server Core, be sure to check out the Ask the Core Team blog.  Excellent stuff!

As the Tool Time columnist at SQL Server Magazine, I’m always on the lookout for great free tools that get on-going support from their creators.  One common experience is finding a nice debugging tool, only to discover that there’s no information on how to interpret the debugger result sets!  (SQLIOSim anyone?)  That’s why I love the Microsoft Advanced Windows Debugging and Troubleshooting blog.  Another must-have on your Favorites list.

Enjoy!

-Kev

Follow me on Twitter

The other cross-thread that had really caught my interest lately is the USA federal governments Open Data Initiative.  I think it’s remarkable that President Obama is the first president to appoint a federal CIO.  (Shouldn’t that have happened in the past?)  In addition, President Obama instructed the entire executive branch to open up their data (where security isn’t at risk) and make it readily available to the public.  And the US government collects mountains of interesting and valuable data for its own uses, but figuring out how or who to share it with was always an afterthought.  While I was a contractor for NASA, for example, I worked on some incredibly interesting projects which yielded amazing and commercially valuable information.  It was all public domain.  But unless you knew it was there, you couldn’t get to it. Making use of all of that data always intrigued me.

Now, with ODI, it’s all being put on the internet at an ever-increasing rate at Data.gov.  However, all of this data, while open and available, is not standardized.  Some data sets might be a CSV file, while others might be something like a spreadsheet.  That means you’ll need to extract, transform, and load that data if you want to synthesize more valuable data sets.

For those reasons, I’ve been researching tools to help make this process easier.  (I also wanted to research SSIS and ETL tools for my Tool Time column in SQL Server Magazine.)  Now, I’ve been following expressor software for quite some time and really like their unique approach.  (I actually ran into the expressor software team at a PASS Summit one or two years ago and asked for a demo of their software.  And I really liked what I saw.)  Rather than the workflow approach used by SSIS, expressor software uses a data mapping approach combined with reusable business rules.  Their mapping approach is fundamentally different from the traditional point-to-point, source-to-target mappings paradigm.  Basically, you can define a semantic type representative of your business data, create a business rule(s) to apply to the data, and then implement a “canonical” mapping which connects data sources and targets to that same semantic type.  And it’s free!

Abstraction is Awesome

What’s cool about that?  Don’t forget that “semantic” means “meaning”.  So a semantic type is an abstraction of the meaning of the data.  The net result is that expressor shields your data integration application, with its associated business and transformation rules, from changes that might occur to underlying target or source files with different field names and data type representations have to be processed. 

For example, let’s assume that you need to process invoices from different vendors in slightly different formats.  If you use a traditional ETL tool like SSIS, any changes in the source and/or target formats will require you to modify your data mappings and transformation rules, because the mappings are tied directly to the metadata structure of the invoice file format(s). expressor, on the other hand, lets you define a common “invoice” semantic type, build all your downstream data processing off that type and map one or multiple invoice file schemas to the type.

This approach greatly simplifies the mapping process and provides for more flexible data integration applications that can be more easily adapted to changes in the source and target data sources.

expressor Studio Desktop

Benefits Abound

Since the semantic types in expressor are captured as reusable artifacts, you can also reuse them again in new data flows within your project(s).  You can even share them across your entire organization.  As I tinkered with the expressor Studio tool, I hit on a few other benefits with this approach:

• Handles data type conversions automatically without having to write data transformation rules for these conversions
• Builds new semantic types from existing types and reuses types in existing and new applications
• Creates multiple, reusable business rules against a single type and applies them repeatedly as needed
• Easily implements data quality rules and constraints

In an Ideal World…

In an ideal world, I’d figure out some brilliant way to make money from bringing together all kinds of that government data that I used to work with.  Other folks are doing it at the Windows Azure Data Market.  But in the meanwhile, I’m also looking forward to tinkering with this data to build better demos.  Along the way, I’m going to use the expressor Studio desktop ETL tool (Did I mention that it’s free!) as well as tell you about my experiences as I try to build out some Data.gov data sets.

Those of you who know me, know that I look a good discussion and cooperative, constructive team work.  So I encourage your feedback and suggestions, as I work through these data integration challenges and share my experiences.  I’m looking forward to sharing with you my insights on what the expressor data integration software can do with this challenge and what some of its features and capabilities are.  In upcoming releases, I’ll let you know what I find intriguing and worth mentioning.

Check out their website, www.expressor-software.com, to learn more about their company and products.

Enjoy,

-Kev

Follow me on Twitter

You can watch the videos now at the URL above or order a DVD, shipped free of charge in the USA and Canada, HERE.

Enjoy!

-Kev

~~~

Twitter at kekline

More content on my Blog