Deployment patterns are of two basic types, regarding database changes that they cover:

• Schema deployment patterns cover any modifications in database schema. They can be divided further into two disjunctive groups:
• Data-independent patterns are applied without requirements to modify existing data. In other words, when I need to apply data-independent pattern, I do not require to affect data being hold in existing tables. They are always deterministic (see below)
  Examples: add/drop views, functions or stored procedures, drop foreign key or column constraint
• Data-dependent patterns require existing data to be modified. The reason for modifications can be, for example, improving data quality before the pattern can be applied.
  Examples: add foreign key, add column constraint
• Data deployment patterns modify purely the data and they keep the schema unchanged. Usually they follow schema deployment patterns in order to insert data into newly introduced tables/columns. They are deterministic or dependent (see below).

Deployment patterns are of different clarity, depending on how straightforward it is to apply the pattern.

• Deterministic patterns include deployment of changes for which there is only one way how to deploy them. Sometimes, the particular pattern can be applied repeatably without negative impact on a database schema and data itself. E.g. deployment of Modify view pattern is pretty straightforward – just execute ALTER VIEW... and you are done. When you execute it twice, it does not cause a problem. Well, modifying the view might break a dependent application, of course, but regarding the database layer, there is neither a risk nor uncertainty in applying it.
• Ambiguous patterns are the ones for which it is not clear how to apply them. For instance, Split column pattern includes a recipe defining the way how to split one column into the two (imagine splitting FullName into FirstName, LastName). How should it be done? Split by spaces? What if FullName includes academic titles? Get rid of them? Make them a part of the FirstName? These questions have to be answered by a developer who knows data quality. Moreover, ambiguous patterns usually depend on existing data and they usually are not repeatable.

Ambiguities are the biggest obstacle for adopting automatic database deployment. If you succeed with reducing the number of ambiguous deployment patterns you will feel less pain in introducing continuous database deployment. There are less ambiguities when deployment patterns are small, e.g. introduce column pattern. On the contrary, when database deployment requires complex patterns (e.g. Introduce Associative Table) to be applied then the number of ambiguities increases.

The main question is: How to reduce the number of ambiguities?

• Deploy database changes often. Then there is smaller chance for introducing complexities between the original and the new database version.
• Use the right tools. Although the existing tools cannot solve all problems connected with database deployment they can provide significant help. They should detect differences between database versions. Consequently, they should assist a developer in classifying database deployment patterns. Of course, the tools should automatically resolve differences for which deterministic patterns can be applied so that the developer can focus on ambiguous problems.

Related posts:

1. Database Deployment Approaches

On the other hand, applications that are oriented strictly on database development (and deployment) follow (as  I call such a development approach) Independent Database Design (IDD). Thus, the major determinant for introducing database schema changes is the need for extending or modifying a nature of the data. This category typically includes data warehouses and high load/performance critical database processes.

It should be clear that IDD is simpler than ADD because a database developer cares just about development and deployment of one layer. But when an application developer takes care of two software layers and their mutual interconnection then he has more complex responsibility. It is natural that deployment of two dependent layers is more complex task than deployment of only one. The situation becomes even more complex when responsibilities for application layer development and database layer development are spread across different teams because of communication issues.

First I focus on IDD because it seems it is simpler than ADD, as if ADD was just extension to IDD. Later in this article I will explain that IID can be significantly different from ADD.

Independent Database Design

If a developer is responsible for development of just a database layer then his task to create deployment strategy is quite straightforward. Once the development phase within an iteration is over he is expected to define a deployment strategy for new version of database schema. The strategy usually is defined in a database deployment script. It includes DDL commands for inserting/modifying/deleting columns in tables and etc. The deployment script often defines also DML commands like inserting initial data to newly created tables. The two general questions come up when thinking about how to define database deployment process:

1. How does the developer identifies what should be deployed into target environment?
2. How does the developer define the deployment process of the database?

The answer to the first question is Database Comparison Tool. There is a number of them in the market. They can compare both database schema and data themselves. When the developer needs to identify what should be deployed he usually works in the following scenario:

1. Compare the target database schema with current development database schema.
2. Identify schema differences
3. Identify which data should be deployed into target database. Database data comparison can help here.

Identifying schema and data differences is quite simple thanks to the mature database comparison tools. The more difficult part come up with the second question. How to define a deployment process? Regardless of whether or not the developer uses some database deployment software the final result is always the same – a deployment SQL script. So, in order to answer the second question related to defining deployment process, the developer has to do the following steps:

1. Create a database schema deployment script. It usually contains only DDL commands.
2. Create database data modification script. It usually contains only DML commands.

Of course, the tools could help here, too. However, the whole process of transforming the identified database differences into the deployment script can be very complex. There can be mutual dependencies among database objects – both schema and data dependencies.

Consider the following example: In a target database there is a table Person with columns Id, FullName. In a development database that I want to deploy to target there is a table Person with columns Id, FirstName, LastName. How should I deploy the development database into target database? Simply with this deployment script?

ALTER TABLE Person ADD COLUMN FirstName VARCHAR(50);
ALTER TABLE Person ADD COLUMN LastName VARCHAR(50);
ALTER TABLE Person DROP COLUMN FullName; -- data gets lost!

Opps! I lost all my data related to person names! Let us try better approach:

ALTER TABLE Person ADD COLUMN FirstName VARCHAR(50);
ALTER TABLE Person ADD COLUMN LastName VARCHAR(50);
UPDATE Person SET FirstName=ExtractFirstName(FullName), ExtractLastName(FullName);
ALTER TABLE Person DROP COLUMN FullName;

Here functions ExtractFirstName, ExtractLastName represent some (potentially complex) transformations defining the process how to retrieve particular parts of the name from its full represennation.

The situation gets more complex if new columns have constraints. For example, FirstName and LastName should not be null. It such a case you are not even able to add a column until you do not specify a default values for the columns being added.

The examples above demonstrates that there is not always a straightforward way how to deploy a new version of database to target environment. If there was then it would be relatively easy to automate creation od a deployment script.

Although nowadays tools can help a developer to pre-generate a deployment script, the developer still is required to manually double-check and fix it if needed. All in all, the whole process of defining a deployment script requires rather a lot of manual work.

Application-Driven Database Design

It might seem that ADD extends IDD approach however it is true only to some extent. In case a developer is creating an application which uses a database only as persistent storage then the database design is driven by application features. If the application stores a property FullName of an entity Person then a table Person with a column FullName has to be modeled in the database.

Later on, when the need for two independent properties FirstName, LastName arises then it has to be reflected in the database as well. Deployment of the application layer is simple since it is stateless – I can simply get rid of the old version of the application that supports only FullName property and replace it completely with new version of the entire application (I simplified it here since I do not count with introducing breaking changes in case other applications depend on the application being deployed). Definition of a deployment script can be simple SQL as in IDD approach.

However, unless the database design is strictly driven by the application then it is possible (at least to some extent) to define a recipe for database deployment directly in the application layer in the language the application layer is written in. Here is an example for migration in Ruby:

class AddFirstNameAndLastNameToPersons < ActiveRecord::Migration
  def change
    add_column :FirstName, :LastName, :string
  end
end

Of course it is just a wrapper that generates good old SQL at deployment time. But with this approach of defining migrations with simple DSL you get greater level of control over deployment process because you can use full power of object-oriented language. You can programmatically combine migrations together, you can programmatically enable/disable them, you can run pre-tests validating that particular migration is even possible to be run, etc.

One of the most popular deployment tools of this kind is Liquibase. Written in Java, it uses special DSL for migrations recipes:

<changeSet id="1" author="bob">
    <addColumn tableName="Person">
        <column name="FirstName" type="varchar(50)"/>
    </addColumn>
    <addColumn tableName="Person">
        <column name="LastName" type="varchar(50)"/>
    </addColumn>
    <sql>UPDATE Person SET FirstName=ExtractFirstName(FullName), ExtractLastName(FullName)</sql>
    <dropColumn tableName="Person" columnName="FullName"/>
    <rollback>
        <addColumn tableName="Person">
            <column name="FullName" type="varchar(100)"/>
        </addColumn>
    <sql>UPDATE Person SET FullName=CreateFullName(FirstName, LastName)</sql>
    <dropColumn tableName="Person" columnName="LastName"/>
    <dropColumn tableName="Person" columnName="FirstName"/>
    </rollback>
</changeSet>

During deployment, the tool is responsible for translating migrations written in XML into SQL of target database server. Liquibase comes with an interesting feature of a rollback deployment in case a business decision to go back to previous version arises. In some cases the rollback does not have to be defined (add column, remove column) since a recipe for rollback can be inherited from the migration definition. But for a more advanced migration like in the example above the rollback recipe has to be defined manually.

Conclusion

Regardless of what approach the development team chooses for database migrations it always results in creating of a database change script. In simple case, the script is first generated by a database comparison tool, then a developer manuall reviews in and corrects conflicting changes. RoundhousE or Liquibase represent such a family of tools.

Advanced tools use custom DSL for defining database migrations. They can be written in application original language (Ruby migrations, FluentMigrator).

There are also mature commercial products that include both database comparison and deployment features in one tool. SQL Data Compare and NeXtep Open Designer are promising tools for database deployment automation.

Related posts:

1. Database Deployment Patterns (Introduction)
2. Database Continuous Integration with DbTracer

We are using Assert.Inconclusive mainly for tests which are:

• Not implemented yet
• Somehow broken or incomplete = requires futher attention
• When test body is for any reason commented out

We are doing this because:

• Inconclusive test can have message
• We want to see such tests in test results on TFS

I did not find any simple way how to make TFS build understand that inconclusive tests should not break the build. Finally,  I found a workaround.  It is based on analysis of trx file after completing test phase of the build. Trx is the XML file containing all details about the test run. Among others, it contains a node <ResultSummary>:

<?xml version="1.0" encoding="UTF-8"?>
<TestRun id="2811e37a-3c6e-4e9f-90a0-6cf0d6e7f44c"
 name="TestResults" runUser="MACHINE\user"
 xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
 ...
 <ResultSummary outcome="Inconclusive">
   <Counters total="487" executed="487" passed="485"
     error="0" failed="0" timeout="0" aborted="0"
     inconclusive="2" passedButRunAborted="0"
     notRunnable="0" notExecuted="0" disconnected="0"
     warning="0" completed="0" inProgress="0" pending="0" />
 </ResultSummary>
 ...
</TestRun>

If there are any inconclusive tests and no other tests failed then the attribute outcome will have value Inconclusive. Analysis of the trx file in the build script was implemented with MsBuld community tasks:

<Target Name="AnalyzeTestResultsFile">
 <ItemGroup>
   <TrxFilePath Include="$(TestResultsRoot)\*.trx" />
 </ItemGroup>
 <!-- Namespace should be specific for MsTest version -->
 <XmlRead Prefix="n"
   Namespace="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"
   XPath="/n:TestRun/n:ResultSummary/@outcome"
   XmlFileName="@(TrxFilePath)">
     <Output TaskParameter="Value" PropertyName="TestOutcome" />
   </XmlRead>
   <Error Text="Test outcome was $(TestOutcome)." Condition="'$(TestOutcome)' == 'Error' or '$(TestOutcome)' == 'Failed' or '$(TestOutcome)' == 'Timeout' or '$(TestOutcome)' == 'Aborted'" />
 </Target>

If you want to have such behavior  in all your builds the the simplest way is to create a file in $(MsBuildExtensionPath) (usually C:\Program Files\MSBuild\):

<?xml version="1.0" encoding="utf-8" ?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\TeamBuild\Microsoft.TeamFoundation.Build.targets"
   Condition=" '$(TeamBuildVersion)' == '' "/>
 <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"
   Condition=" '$(MSBuildCommunityTasksLib)' == '' " />
 
 <PropertyGroup>
   <TestConfigurationDependsOn>
     ResetTestWorkspace;
     $(TestConfigurationDependsOn);
     AnalyzeTestResultsFile;
   </TestConfigurationDependsOn>
 </PropertyGroup>
 
 <Target Name="ResetTestWorkspace">
   <RemoveDir Directories="$(TestResultsRoot)" />    
   <PropertyGroup>
     <StopOnTestFailure>false</StopOnTestFailure>
   </PropertyGroup>
 </Target>
 
 <Target Name="AnalyzeTestResultsFile">
   <ItemGroup>
     <TrxFilePath Include="$(TestResultsRoot)\*.trx" />
   </ItemGroup>
   <!-- Namespace should be specific for MsTest version -->
   <XmlRead Prefix="n"
     Namespace="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"
     XPath="/n:TestRun/n:ResultSummary/@outcome"
     XmlFileName="@(TrxFilePath)">
     <Output TaskParameter="Value" PropertyName="TestOutcome" />
   </XmlRead>
   <Error Text="Test outcome was $(TestOutcome)." Condition="'$(TestOutcome)' == 'Error' or '$(TestOutcome)' == 'Failed' or '$(TestOutcome)' == 'Timeout' or '$(TestOutcome)' == 'Aborted'" />
 </Target>
</Project>

I redefined property $(TestConfigurationDependsOn) so that target ResetTestWorkspace is automatically called right before test run is invoked. Target AnalyzeTestResultsFile is called after test run when trx file is already available.

With this TFS template all your builds will not fail when you have some tests inconclusive.

No related posts.

Promiscuous Women Cause Problems for Database Administrators

This is a fiction story that never happened:

Juliet Capulet meets Paul Smith; they fell in love and marry on 25th August 2009. When Juliet is back at work she asks the db admin to change her name in a company database. The admin is not happy to have additional work with this stuff but he does it for her.

Juliet’s and Paul’s marriage is not going well and Juliet divorces Paul after a short period. Soon Juliet feels alone, she meets Romeo Montague and, of course, falls in desperate love with him. Logically, another wedding is arranged and Juliet asks db admin to change her name again on 10th January 2010.

The tragedy of all Juliet’s love relationships finishes when she finds her husband Romeo as a roulette player spending all their money. She divorces again and changes her name to Juliet Singleton on 21st July 2010 with the vision not to marry anybody again. Needless to say, the db admin is close to committing suicide because his database does not allow changing the name more than twice.

What would you suggest the database admin to do better in his next life?

Problem description

Let us summarize what the db admin was asked to do with Juliet’s personal data:

• Juliet Capulet gets the job on 2005-03-01
• Name Juliet Capulet changes to Juliet Smith on 2009-08-25
• Name Juliet Smith changes to Juliet Montague on 2010-01-10
• Name Juliet Montague changes to Julie Singleton on 2010-07-21

Solution No.1: Roll off Out-dated data

The simplest solution for keeping historical data in the database is to create a duplicated record before update. After that we can update the requested data. In case of tracing Juliet’s name we get these data in the database table:

In real database schema, id is primary key and has to be unique. So we need extra column containing information about row id from which data were rolled off. Additionally, it would be helpful to have a column representing status of the record. Because situation gets more complex here I show how data evolve in database:

Juliet Capulet gets the job on 2005-03-01.

Name Juliet Capulet changes to Juliet Smith on 2009-08-25.

Name Juliet Smith changes to Juliet Montague on 2010-01-10.

Name Juliet Montague changes to Julie Singleton on 2010-07-21.

You can see that each change to the record with id=124 invokes a) creating duplicated row with status=history and b) updating the column name to a requested value with created_date set to current date. When the active record is going to be “deleted” then only the status flag changes the value from active to history.

Advantages

It is simple to implement audit logging based on rolling out old data. You do not need additional tables. Maintenance is simple, too. If you decide to remove old data then it is the matter of one SQL.

Disadvantages

But what if you need to implement audit logging in more tables? You need to add columns original_id, status, created_date and created_by. And you have to implement previously introduced roll-off logic, either to database directly (usually with table triggers) or in the application.

The problem can be that when data changes then the whole record is copied, i.e. also data which do not change. It causes data duplication and, potentially, database disk space can increase markedly if changes occur frequently. For example, if table persons has a column photo with binary data containing the photography then each and every time Juliet’s name changes the whole record (including the photo) is copied to rolled-off record.

Another disadvantage is that the complexity of each table supporting audit logging increases. You must have in mind all the time that retrieving the records is not simple. You always have to use the SELECT clause with condition

SELECT * FROM persons WHERE STATUS='active'

When you need to roll back historical data for a particular record you have to copy all columns from the historical record to the active record. It would be easier just to switch status value from “history” to “active” but it would break reference integrity if the record was referenced in other tables.

Solution No.2: Dedicated Data-Tracing Table

Another approach is based on separate audit log table that is dedicated to logging data changes coming from all tables having audit log feature:

Advantages

This solutions separates the concerns much better. There is only one common data store for all historical data. Further, you have to implement audit log feature only once for all tables which should support it.

Compared to the previous solution, when the record changes only the changed data are logged. Thus it is expected to be a little bit faster but I did not do any tests so I cannot prove it exactly. The real advantage is that you need lower disk space to store data changes. You can improve the table structure when you normalize the log table and you put just integer references to “table” and “column”.

You can easily retrieve user activity just by the query

SELECT * FROM audit_log WHERE changed_by='admin'

Disadvantages

All data changes are logged in a common table so old and new values stored there must be of some generic type. For example, in MS SQL it can be “sql_variant”. Or you can convert the values to “varchar” before storing them in the log table.

Rollback is rather difficult task to be implemented. If you want to roll data back to 3rd March 21010 you have to use query

SELECT [TABLE], [COLUMN], old_value, new_value FROM audit_log WHERE [TABLE]='persons' AND ROW=124 AND changed_date > '2010-03-01' ORDER BY changed_date

and then you iterate through the result set and build the dynamic query because table and column info is in the result. It is more complicated compared to the previous solution. However, the feature has to be implemented only once and then it is reused across the whole application.

No related posts.

TypeText(string) is slow because during run it fires sequences of key events: KeyDown(char), KeyPress(char), KeyUp(char). The sequence is invoked as many times as the length of the input string is. As the result, the input text is typed to the text field one by one char. It is slow and when you have hundreds of tests then you wait tens of minutes for their execution! I created simple extension method to speed up typing of text markedly – for a string 10 chars long the performance of typing is around 30 times higher.

public static class WatiNExtensions
{
    public static void TypeTextQuickly(this TextField textField, string text)
    {
        textField.SetAttributeValue("value", text);
    }
}

In my solution I do not use simulated key events. I use direct setting of value attribute in text field. The limitation of this solution is that it does not raise focus and blur events. Additionally, no key events are fired.

Usage is simple:

var browser = new IE("http://google.com");
browser.TextField("q").TypeTextQuickly("WatiN"); // fast - raises neither key events nor focus and blur on text field
// browser.TextField("q").TypeText("WatiN"); // slow - raises key events

No related posts.

C:\Program Files\Git\bin>git svn init -s file:///D:/path/to/svn/repository
Initialized empty Git repository in C:/Program Files/Git/bin/.git/

But Ooops! Error occurred when I tried to fetch all the history to Git repository:

C:\Program Files\Git\bin>git svn fetch
 A       notes.txt
 A       pages.txt
 A       settings.config
r7 = f3dba0984f11e2f358923c23d89440789eb4591b (refs/remotes/references)
Permission denied: Can't open '/tmp/report.tmp': Permission denied at C:\Program Files\Git/libexec/git-core/git-svn line 2723

It was an misguiding error since I had neither a “tmp” directory in the original svn repository nor “report.tmp” directory. For my surprise the problem was caused by different formats used by in original Subversion repository and by Subversion module included in my Git instance that probably uses older Subversion format. I use Git 1.6.5.1.1367.gcd48 that probably has Subversion module supporting format 4 but my Subversion repository was in format 5.

Solution

Luckily the solution for such WTF exists. If I access Subversion repository via SVN server instead of directly via file path problem will not occur. Thus, at first it is necessary to run SVN server:

svnserve.exe -d -r D:/parent/path/to/repository/

And then you can access Subversion repository via its server:

git.exe svn init -s "svn://localhost/relative/path/to/subversion/repository"

No related posts.

The common repository for all source code changes is Subversion (currently we think of migrating to Git, anyway). For tracking changes of database schema changes we use Subversion, too.

In our team each developer has his own database server installed on his PC and he is an administrator of his own database server. The core of everything is the SCM. Developers check out latest code from there, integration server uses it as well to get code base and integration tests. The idea connected with tracking changes of database schema was to include changes under version control. Then both developers and integration server were able to retrieve database schema changes and to handle them.

Database Change Scripts

If a developer wishes to do some changes in database schema, he can freely do that on his local database without fear to influence work of the others. As the result of his work he is expected to create a database change script that includes all schema changes needed to be implemented. Let’s say we need to add a column to the table. The change script will contain these SQL commands:

ALTER TABLE ORDER
ADD STATUS INT NOT NULL DEFAULT 0
GO

The script will add a column “Status” to a table “Order”. You can feel it would be very annoying for a developer to change database schema only via change scripts in regular daily work. We established a helpful practice how to make things easier for the developer. The change script should be the result of his work that he wish to be integrated. We work as follows:

1. When developer is going to do schema changes (on his local machine, of course), first he creates a clone of database version that was already fully integrated to source control. Let’s say that database is in version 1.5 so he creates a clone of it with help of DbTracer:

   cmd> dbt create /database=reference_db_v_1_5 /db-version=1.5 /connection=(local)

2. At this point he has a reference database that will serve him as a reference point when he will compare it with his test database. The test database is created in the same way as reference one:

   cmd> dbt create /database=test /connection=(local) /db-version=1.5 /override-if-exists

3. Now the developer is ready to do whatever he wants in the “test” database. For example he can add a column “Status” to a table “Order”. Of course you are not limited to add only one schema change to one change script.
4. When he is ready with schema changes he compares the test database with a reference database “reference_db_v_1_5″. Tools for comparing databases are very much helpful here. In my team we use EMS DB Comparer.
5. The tool generates a change script. It is possible to modify it manually if we are not satisfied with the result. After that, the change script is ready to be checked in to version control.

Change Script Names

We use the following convention for our change scripts:

<major_version>.<minor_version>.<revision>-<some description of change script>.UP.sql

For example:

003.012.0541-add column Status to table Order.UP.sql

In this example the change script lies within database major version=3, minor version=12 and its revision=541. There is also a short description “add column Status to table Order” defining what is inside the change script. Additionally, the keyword “UP” in the change script is reserved for future when we will need to distinguish between upgrade change scripts and downgrade change scripts.

Now the change script is ready to go to version control.

Our Solution – DbTracer

We searched for a simple tool that would maintain a our database migration scripts. Additionally, it should provide following features:

• command line interface
• storing SQL database version in the database itself

Unfortunately, no such tool existed that would have been acceptable. Hence, we inspired by Alexander Kleshchevnikov and wrote our own tool DbTracer. Currently it is in experimental phase for public. We use it internally in our team but still API changes are going on.

Further Reading

• Three Rules for Database Work
• Database Continuous Integration in .NET
• Evolutionary Database Design

Related posts:

1. Database Deployment Approaches