Ed Vassie, the creator of FineBuild, has revised and improved the tools since I last wrote about it.  Ed has naturally added a number of bug fixes and documentation improvements. Several new features… [READ MORE]

I know that I did just that, for years.  It wasn’t until I was trying to performance tune some existing code that I realized I hadn’t actually taken caching of data and execution plans into account.  So all those modified stored procedures that I was so proud of might not actually be faster than the first generation of procedures because I hadn’t checked to ensure that I was testing cached programs against uncached programs (and, by extension, the data used by those programs).  That’s easy enough to fix with a test harness.  Test harness were originally an actual, physical harness used by engineers to clamp down parts of an electrical or mechanical device they were prototyping.  Ours is no different.  It locks down all of the assumptions about our code (like my early, false assumption that I didn’t need to clear the caches) and adds a metric or two for good measure – literally – so we can better measure what’s happening in that code.

Here’s what my test harness looks like:

/* Transact-SQL test harness by Kevin Kline, http://KevinEKline.com, Twitter at kekline */

/* Flush dirty pages from the buffer to the database files. */

CHECKPOINT;

/* Flush the data cache and procedure cache, respectively. For DEV environments only! */

DBCC DROPCLEANBUFFERS;

DBCC FREEPROCCACHE;

/* Enable statistics tracking for IO and timings. Remember, SET commands remain enabled during a session until disabled. */

SET STATISTICS IO ON;

SET STATISTICS TIME ON;

-- Whatever SQL code you'd like to process goes below.

SELECT SalesOrderID

FROM Sales.SalesOrderHeader H

WHERE CustomerID = 344

GO

SET STATISTICS IO OFF;

SET STATISTICS TIME OFF;

/* Textual Execution Plans, if desired.

SET SHOWPLAN_TEXT ON;

SET SHOWPLAN_TEXT OFF;

*/

I also like to include the execution plans a lot of the time.  You might wonder why I don’t save the execution plans for the GUI in SSMS?  Well, I’m a big advocate of scripting in general because I like to automate activities.  By pulling the execution plans using scripts, I can use SQLCMD to schedule a large number of query executions during the evening and have the results ready for analysis when I come back into the office in the morning.  Workin’ smarter, not harder, Baby!

So how does this test harness work for you?  Do you use other elements in yours?  If so, share your experiences here!

Thanks,

-Kevin

-Follow me on Twitter

So I’m going to start sharing what I’m reading at least once per week, partly so that I don’t firehose too many reading links directly into your brain (where I to do it say once per month) and partly to solidify in my own mind the information that I’m reviewing. So here are a few good links for the seven days leading up to July 22, 2001:

• Microsoft and Whitehouse partnership on BigData: BigData isn’t a particularly new concept.  But I was intrigued to learn that the National Science Foundation, Microsoft, and 13 other teams were partnering on developing better BigData analytics for lots of government data from activities such as healthcare, economic development, education, transportation, and the power grid.  Cools stuff!  Plus, Microsoft has developed a new tool called Project Daytona to better harness the power of the cloud, in general, and Windows Azure, specifically.
• While we’re on the topic of Federal IT in the Cloud be sure to read this linked article from ComputerWorld.  Say what you will about our government, but putting government IT in the cloud and increasing both its transparency and availability will make a huge difference in how the Federal government will be able to service the public.  We’re talking as big a difference as corporations experienced between the “catalog on the web” experience of the 1990′s to the Web2.0 experience of today.
• If you’re the social media type, give this article a read discussing the Power of Hashtags in Social Media.
• The Register, of the UK, whose tagline is “Biting the hand that feeds IT” has a great article on a spat over database technologies between the IT sage Michael Stonebreaker and Google.  It’s a great read if for no other reason than to prove that databases are worth fighting over.
• And if you think Microsoft is still towing the relational database barge without thinking about other technologies, you need to read up on Projects Dryad and Daytona.
• Finally, I’m still getting lots of questions about when and where to limit SQL Server’s Max Degrees of Parallelism.  Be sure to read Microsoft’s Recommendations and Guidelines for ‘max degree of parallelism’ configuration option here.

And just because so many of us in IT are closet or former musicians, there’s Live Guitar Lessons with Steven Krenz, sponsored by my hometown boyz at Gibson Guitar.

Got a favorite article or tool tip? Let me know!  Enjoy,

-Kev

Follow me on Twitter.

Remote Desktop Services Component Architecture Poster

Grab your own poster! A visual guide to key Remote Desktop Services technologies in Windows Server 2008R2

Virtually Free

Get the latest update rollup package for the Hyper-V role in Windows Server 2008 R2 and be sure to bookmark the Windows Virtualization Team Blog.  Be sure to check out blog World Simplified is a Virtual World. And doncha evah neglect application virtualization, such as the goodness at the App-V Product Team Blog.

Let’s Optimize Some Desktops (Assuming You Have Gone Full Cloud Yet)

Check out The Official Microsoft Desktop Optimization Pack (MDOP) Blog where you can get cool tools like the Diagnostics and Recovery Toolset (DART), currently in its v7 beta release.

We Can Always Perform a Little Better

Have you checked out the Ask the Performance Team blog yet.  You should!  Then there’s the Microsoft Enterprise Networking Team blog.  And if you’re looking for help with the Windows Server Core, be sure to check out the Ask the Core Team blog.  Excellent stuff!

As the Tool Time columnist at SQL Server Magazine, I’m always on the lookout for great free tools that get on-going support from their creators.  One common experience is finding a nice debugging tool, only to discover that there’s no information on how to interpret the debugger result sets!  (SQLIOSim anyone?)  That’s why I love the Microsoft Advanced Windows Debugging and Troubleshooting blog.  Another must-have on your Favorites list.

Enjoy!

-Kev

Follow me on Twitter

The other cross-thread that had really caught my interest lately is the USA federal governments Open Data Initiative.  I think it’s remarkable that President Obama is the first president to appoint a federal CIO.  (Shouldn’t that have happened in the past?)  In addition, President Obama instructed the entire executive branch to open up their data (where security isn’t at risk) and make it readily available to the public.  And the US government collects mountains of interesting and valuable data for its own uses, but figuring out how or who to share it with was always an afterthought.  While I was a contractor for NASA, for example, I worked on some incredibly interesting projects which yielded amazing and commercially valuable information.  It was all public domain.  But unless you knew it was there, you couldn’t get to it. Making use of all of that data always intrigued me.

Now, with ODI, it’s all being put on the internet at an ever-increasing rate at Data.gov.  However, all of this data, while open and available, is not standardized.  Some data sets might be a CSV file, while others might be something like a spreadsheet.  That means you’ll need to extract, transform, and load that data if you want to synthesize more valuable data sets.

For those reasons, I’ve been researching tools to help make this process easier.  (I also wanted to research SSIS and ETL tools for my Tool Time column in SQL Server Magazine.)  Now, I’ve been following expressor software for quite some time and really like their unique approach.  (I actually ran into the expressor software team at a PASS Summit one or two years ago and asked for a demo of their software.  And I really liked what I saw.)  Rather than the workflow approach used by SSIS, expressor software uses a data mapping approach combined with reusable business rules.  Their mapping approach is fundamentally different from the traditional point-to-point, source-to-target mappings paradigm.  Basically, you can define a semantic type representative of your business data, create a business rule(s) to apply to the data, and then implement a “canonical” mapping which connects data sources and targets to that same semantic type.  And it’s free!

Abstraction is Awesome

What’s cool about that?  Don’t forget that “semantic” means “meaning”.  So a semantic type is an abstraction of the meaning of the data.  The net result is that expressor shields your data integration application, with its associated business and transformation rules, from changes that might occur to underlying target or source files with different field names and data type representations have to be processed. 

For example, let’s assume that you need to process invoices from different vendors in slightly different formats.  If you use a traditional ETL tool like SSIS, any changes in the source and/or target formats will require you to modify your data mappings and transformation rules, because the mappings are tied directly to the metadata structure of the invoice file format(s). expressor, on the other hand, lets you define a common “invoice” semantic type, build all your downstream data processing off that type and map one or multiple invoice file schemas to the type.

This approach greatly simplifies the mapping process and provides for more flexible data integration applications that can be more easily adapted to changes in the source and target data sources.

expressor Studio Desktop

Benefits Abound

Since the semantic types in expressor are captured as reusable artifacts, you can also reuse them again in new data flows within your project(s).  You can even share them across your entire organization.  As I tinkered with the expressor Studio tool, I hit on a few other benefits with this approach:

• Handles data type conversions automatically without having to write data transformation rules for these conversions
• Builds new semantic types from existing types and reuses types in existing and new applications
• Creates multiple, reusable business rules against a single type and applies them repeatedly as needed
• Easily implements data quality rules and constraints

In an Ideal World…

In an ideal world, I’d figure out some brilliant way to make money from bringing together all kinds of that government data that I used to work with.  Other folks are doing it at the Windows Azure Data Market.  But in the meanwhile, I’m also looking forward to tinkering with this data to build better demos.  Along the way, I’m going to use the expressor Studio desktop ETL tool (Did I mention that it’s free!) as well as tell you about my experiences as I try to build out some Data.gov data sets.

Those of you who know me, know that I look a good discussion and cooperative, constructive team work.  So I encourage your feedback and suggestions, as I work through these data integration challenges and share my experiences.  I’m looking forward to sharing with you my insights on what the expressor data integration software can do with this challenge and what some of its features and capabilities are.  In upcoming releases, I’ll let you know what I find intriguing and worth mentioning.

Check out their website, www.expressor-software.com, to learn more about their company and products.

Enjoy,

-Kev

Follow me on Twitter

As Mladen says:

…this release adds support for SQL Server 2011 (Denali) CTP1 and fixes a few bugs. Because of the new SSMS shell in SQL 2011 CTP1 the SSMS Tools Pack 1.9.4 doesn’t have regions and debug sections functionality for now. The fixed bugs are: A bug that prevented to create insert statements for a database A bug that didn’t script commas as decimal points correctly for non US settings….

Enjoy!

-Kev

Here are a hand full of cool migration tools worth mentioning:

Microsoft SQL Server Migration Assistant (SSMA) for MySQL: Migrate from MySQL to SQL Azure or SQL Server with ease.  Plus, here are a few related Knowledge Base (KB) articles:

Azure SQL Service: Database Migration To Sql Azure
http://support.microsoft.com/kb/2212855/en-US

Azure SQL Service: Database Migration To Sql Azure
http://support.microsoft.com/kb/2212990/en-US

Azure SQL Service: Developing Applications For Sql Azure
http://support.microsoft.com/kb/2213496/en-US

SQL Server Migration Assistant (SSMA) for Sybase: Migrate from Sybase ASE to SQL Server 2005, SQL Server 2008 and SQL Server 2008 R2.

SQL Server Migration Assistant (SSMA) for Microsoft Access: Migrate from Access to SQL Server 2005, SQL Server 2008, SQL Server 2008 R2 and SQL Azure.

SQL Server Migration Assistant (SSMA) for Oracle: Migrate from Oracle to SQL Server 2005, SQL Server 2008 or SQL Server 2008 R2.  I’m thinking about installing it on my SQL Servers even without even needing to migrate existing Oracle databases to SQL Server.  Why?  Well as an old Oracle hand, I came to really enjoy quite a few Oracle PL/SQL system packages (kind’a like a SQL Server system stored procedure, but often more powerful).  As it turns out SSMA-Oracle includes stored procedures, extended stored procedures, and CLR routines that reproduce the functionality in most all of the cool and powerful Oracle packages like DBMS_PIPES. It’d be nice to have those on my SQL Servers just because I know them and like them.

Microsoft Drivers for PHP for SQL Server 2.0: Microsoft Drivers for PHP for SQL Server – and PHP is good.

Microsoft Services  for Mission Critical Customers: Many enterprise customers running mission critical applications on SQL Server have asked for more – more service and support for their environments. This is an add-on that costs extra, but it’s worth it for those running the systems that keep the company in business.
If you’ve tried any of these tools out, I’m keen to hear your experiences.  Did they work well for you?  Did they work, though poorly?  Did they fail utterly?  Inquiring minds want to know.

Another tool I really like is PowerWF.  PowerWF is a really cool visual workflow builder that creates PowerShell scripts for you.  Turns out that it integrates with PowerGUI!

This video shows 2 different ways that PowerWF Workflows can be run from within Quest Software’s PowerGUI tool.

Watch the Video - Download PowerWF

Enjoy!

-Kev

I was just bragging about how Toad for SQL Server keeps getting better.  In that post, I also pointed out a lot of great resources you can put to work immediately on improving your skills with this great tool.  (Incidentally, there’s a freeware version without all of the features, but it’s still quite useful.  And you can always use the beta product, if you want all of the features and many new features that are undergoing community testing.)

Ain't he handsome?

One of the reasons that Toad is so good is that it’s always been a community-driven product.  Back when I used Oracle every day, TOAD was an acronym = Total Oracle Application Development.  It didn’t take long for Toad to rise above the acronym transform into the eponymous term denoting “kick-butt database tool” just a few years before Toad began to go cross-platform.  Now that Toad is solidly cross-platform with versions for DB2, MySQL, and Cloud to boot, it’s worth pointing out that Toad got to be what it is today entirely from community feedback.  Back in the day, when I worked in Quest’s R&D team, the developers literally kept a checklist of cool suggestions from the community and worked against that to develop new features.  My point isn’t to fully describe the inner workings of the Toad dev team, rather I wanted to highlight how incredibly important community feedback is to this tool and the developers behind it.

To further that point, there’s a new community for all users of Toad for SQL Server.  Here, you can:

• get solutions and ask questions
• watch training and “tips & tricks” videos
• share your knowledge and experience
• get the latest information on new Toad for SQL Server features and enhancements
• download the latest releases
• get expert tips and techniques
• plus, directly communicate with the developers

Look around, make yourself at home!

Enjoy,

-Kev

Follow me on Twitter