Branching Case Study - Using the '-i' flag

by Robert Cowham, Perforce Consulting Partner

Introduction

This article shows how to use some of the more advanced branching features in Perforce such as the "integrate -i" to recreate a branching structure from a rather messy starting point.

There is an introductory branching article which you should read first which gives an overview of the integrate command.

The actual examples of commands etc make use of standard Unix shell features and text processing commands with which the reader is assumed to be fairly familiar.

In particular I was using the Bash shell for its excellent text processing utilities.

Background

The client had a web site that had been in production for several years and the web site consisted of a mixture of HTML, Perl scripts, graphics, etc.

Perforce had been in use for some time in a purely development capacity and was being used in basic check-in/check-out mode using a single mainline. Modified HTML pages, scripts etc were transferred to test and live servers using scripts (this in itself was an advance on the previous mode of working where people manually transferred stuff on to the live server or made changes there!).

A quick analysis showed the following statistics:

• 30,000 items on the live server
• 8,000 items on the development machine that were actively being controlled using Perforce (and thus supposedly being "promoted" to live from Perforce)

The obvious implication being that there were 22,000 items not under control!

End Goal

The basic principle we wanted to achieve was that everything on the production server should be taken from the Perforce repository and thus we could always see its complete history (including changes).

The desired process that we wanted to support was fairly standard for a web site. Create 3 branches: Main, Test and Live. Most development happens on Main, and changes are checked in there. At regular intervals, changes are propagated to the Test branch for QA testing. When that looks OK, changes are propagated to Live.

See the diagram below.

Note that sometimes QA testing can throw up problems which are fixed and then propagated back to Main.

Thus we wanted to create the above structure from a starting point of only 8,000 files currently in Main.

Detailed Analysis

The analysis above shows that we had a lot of files that were not in Perforce at all. In fact things were a bit more complicated.

We discovered the following types of files:

1. Files on the production server that weren't in Perforce at all (22,000)
2. Files on the production server (Live files) that were in Perforce, split as follows:
1. Live files which were the same as the head revision in Perforce
2. Live files which were the same as a previous revision in Perforce
3. Live files which were not the same as any revision in Perforce!

There were also some Live files such as configuration files with the names of machines, or IP addresses which needed to be different in the Live environment.

Steps

The actual process of importing files in to Perforce and creating the desired branching relationships, together with appropriate analysis to help along the way, was done using the steps set out below.

The main areas of the depot were (ignoring extra path components present on the actual customer site):

Import Snapshot of Live

We took a copy of the live server to a directory, and created a client workspace with a simple mapping to import everything into //depot/snap/...

We then used "p4 add" to add all the files into Perforce:

find . -type f -print | p4 -x - add

and submitted the result.

Validating Snapshot

One of the developers spent a some time going through all the files which were now in Perforce and deleting files (p4 delete) which were obviously not in use and had just accreted over the years on the live server. This included files such as *.bak, *.tmp, test_*, etc.

It was important to do this after importing so that we could get them back if we made a mistake.

Live Files Same As Main

For files in Snap which were the same as the head revision of Main we just wanted to branch the file from Main into Test.

In the diagram, Snap#1 is the same as Main#3 so we just integrate Main#3 to Test#1 (and there is no direct branching relationship between Main and Snap).

In order to do this we needed to find this set of files (using diff2), and then do the integration to create the new branch.

The commands were:

$ p4 diff2 -ds //depot/snap/... //depot/main/...
	> snap-main-diff2.out

$ cat snap-main-diff2.out

==== //depot/snap/jam/headers.c#1 (xtext) -
//depot/main/jam/headers.c#1 (xtext) ==== identical
==== //depot/snap/jam/headers.h#1 (xtext) -
//depot/main/jam/headers.h#1 (xtext) ==== identical
==== //depot/snap/jam/jam.c#1 (xtext) -
//depot/main/jam/jam.c#2 (xtext) ==== content
add 1 chunks 1 lines
deleted 0 chunks 0 lines
changed 0 chunks 0 / 0 lines
==== //depot/snap/p4-doc/notes/note1.txt#1 -
  <none> ===
==== //depot/snap/p4-doc/notes/note2.txt#1 -
  <none> ===

This produces a file listing in which all identical files between the two areas of the repository are identified with a "==== identical" at the end of the line. We can then use that to branch those files directly from Main to Test.

$ grep "==== identical" snap-main-diff2.out |
 cut -d' ' -f5 |
 cut -d# -f1 |
 while read m ; do
   t=${m/main/test} ;
   p4 integ $m $t ;
 done

//depot/test/jam/Jam.html#1 - branch/sync from
	//depot/main/jam/Jam.html#1
//depot/test/jam/Jambase.html#1 - branch/sync from
	//depot/main/jam/Jambase.html#1
//depot/test/jam/Jamfile.html#1 - branch/sync from
	//depot/main/jam/Jamfile.html#1

$ p4 submit

Note that this set of commands relies on there not being a space (' ') character in the filenames (exercise for the reader if so!).

Live Files Not In Main

These are files in the snapshot of the live server but not previously in the Main branch (which in our case was nearly 22,000 files - i.e. most of them).

We wanted to do 2 things: integrate from Snap to Main and then integrate those files from Main to Test immediately, to create a clean integration relationship between the branches Main and Test.

So Snap#1 is integrated to become Main#1 which is then integrated to become Test#1.

The commands were:

$ p4 integ //depot/snap/... //depot/main/... >
	integ-snap-main.out

$ cat integ-snap-main.out

//depot/main/jam/Jam.html - can't integrate from
  //depot/snap/jam/Jam.html#1 without -i flag
//depot/main/jam/Jambase.html - can't integrate from
  //depot/snap/jam/Jambase.html#1 without -i flag
//depot/main/p4-doc/notes/note1.txt#1 - branch/sync from
  //depot/snap/p4-doc/notes/note1.txt#1
//depot/main/p4-doc/notes/note2.txt#1 - branch/sync from
  //depot/snap/p4-doc/notes/note2.txt#1

$ p4 submit

The integ-snap-main.out contains error messages of the form "can't integrate from <filename> without -i flag" which we can use to detect which files are already present in Main. In this instance we want to submit the files that we could integrate and then use the -v flag on grep to ignore files matching the comment, so that we can branch the other ones immediately.

$ grep -v "can't integrate from" integ-snap-main.out |
   cut -d# -f1 |
   while read m ; do
     t=${m/main/test} ;
     p4 integ $m $t ;
   done

//depot/test/www/pitch/page06.html#1 - branch/sync from
  //depot/main/www/pitch/page06.html#1
//depot/test/www/pitch/page07.html#1 - branch/sync from
  //depot/main/www/pitch/page07.html#1
//depot/test/www/pitch/page08.html#1 - branch/sync from
  //depot/main/www/pitch/page08.html#1

$p4 submit

Live Files Same As Non Head Revision of Main

There were about 200 such files that existed in Snap, and for which there was an entry in Main, but the version in Snap was the same as an older revision in Main, rather than the head revision.

For example, //depot/snap/abc.html#1 was the same as //depot/main/abc.html#3 and the head revision was #5.

First we had to find these files (which involved a Perl script differ.pl). Then we had to set up appropriate integration history.

In the example of abc.html we did:

$ p4 integ //depot/main/abc.html#3 //depot/test/abc.html
$ p4 submit

This branches Test from the appropriate revision of Main - any later changes on Main would be propagated by the normal integrate from Main and Test at the appropriate time.

Note that in this instance there is no branching relationship between the files in Snap and the corresponding files in Main but the contents of Snap#1, Main#3 and Test#1 are all the same.

So detailed steps were:

$ p4 diff2 -q //depot/main/... //depot/snap/... |
    grep '=== content' > diffs.out

$ cat diffs.out

==== //depot/main/jam/jam.c#2 (xtext) -
  //depot/snap/jam/jam.c#1 (xtext) ==== content
==== //depot/main/jam/jam.h#2 (xtext) -
  //depot/snap/jam/jam.h#1 (xtext) ==== content

$ ./differ.pl diffs.out

No revision: //depot/main/jam/fred.txt#2
Identical: //depot/main/jam/jam.c#1
Identical: //depot/main/jam/jam.h#1

In the above instance, version #1 of the files is identical whereas #2 is the current head revision, so we just integrate from #1.

$ grep "^Identical" differ.out |
    cut -d' ' -f2 |
    while read m ; do
      t=${m/main/test} ;
      t=`echo $t | cut -d# -f1` ;
      p4 integ $m $t ;
    done

//depot/test/jam/jam.c#1 - branch/sync from
  //depot/main/jam/jam.c#1
//depot/test/jam/jam.h#1 - branch/sync from
  //depot/main/jam/jam.h#1

$ p4 submit

Live Files Not Same As Any Revision of Main

This is similar to the above case, but there were actually no revisions in Main exactly the same as the revision in Snap (discovered by our Perl script).

In the example of abc.html we did:

$ p4 integ -i //depot/snap/abc.html //depot/main/abc.html
$ p4 resolve -at
$ p4 submit

This overwrites the file in Main with the version from Snap to create #6. Note that the "-i" flag was needed because Snap and Main files are not directly related at this point.

We then branch this version to Test, and then "back out" the change on Main, which means take out #5 and check in again as #7 (see Tech Note 14 for more information about this).

To back out the change on Main and restore the previous head revision:

$ p4 sync //depot/main/abc.html#5
$ p4 edit //depot/main/abc.html
$ p4 sync
$ p4 resolve -ay
$ p4 submit

The result of this step is that the last version we had in Main (before doing anything) would be propagated to Test the next time an integration was done between the two branches.

Note that it was important that we actually check the contents of each of these files and decide that this was an appropriate thing to do. This involved a couple of days discussing the detailed list of files with the developers.

Detailed Steps

Recall the output from our script:

$ ./differ.pl diffs.out

No revision: //depot/main/jam/fred.txt#2
Identical: //depot/main/jam/jam.c#1

So files are identified with "No Revision" if they are the ones we want. Thus the first thing is to integrate everything from Snap to Main (overwriting Main).

$ grep "^No revision" differ.out |
    cut -d' ' -f3 |
    cut -d# -f1 |
    while read m ; do
      s=${m/main/snap} ;
      p4 integ -i $s $m ;
    done

//depot/main/jam/fred.txt#2 - integrate from
  //depot/snap/jam/fred.txt#1

$ p4 resolve -at

/home/robert/work/main/jam/fred.txt - vs
  //depot/snap/jam/fred.txt#1
//article/main/jam/fred.txt - copy from
  //depot/snap/jam/fred.txt

$ p4 submit

Now we integrate files just changed in Main to Test.

$ grep "^No revision" differ.out |
    cut -d' ' -f3 |
    cut -d# -f1 |
    while read m ; do
      t=${m/main/test} ;
      p4 integ $m $t ;
    done

//depot/test/jam/fred.txt#1 - branch/sync from
  //depot/main/jam/fred.txt#1,#3

$ p4 submit

And then we restore previous head revisions in Main.

$ grep "^No revision" differ.out |
    cut -d' ' -f3 |
    while read m ; do
      p4 sync $m ;
      m=`echo $m|cut -d# -f1` ;
      p4 edit $m ;
      p4 sync ;
    done

//depot/main/jam/fred.txt#2 - updating
   /home/robert/work/main/jam/fred.txt
//depot/main/jam/fred.txt#2 - opened for edit
... //depot/main/jam/fred.txt - must sync/resolve #3
  before submitting
//depot/main/jam/fred.txt#3 - is opened and
  not being changed
... //depot/main/jam/fred.txt - must resolve #3 before
  submitting

$ p4 resolve -ay

/home/robert/work/main/jam/fred.txt - vs
  //depot/main/jam/fred.txt#3
//article/main/jam/fred.txt - ignored
  //depot/main/jam/fred.txt

$ p4 submit

Extra Complications

Some of the above was complicated by the fact that there were files in Snap which differed from the file in Main only by filetype (e.g. text+x instead of text) - see "p4 help filetypes". We took a list of those and decided which type was correct and fixed in the appropriate branch.

The Result

The result that we wanted was for every file in Snap to be in Test (tested this by running "p4 diff2 -q //depot/snap/... //depot/test/...") and with every file in Test be branched from a file in Main (tested by running tested this by running "p4 integ -n //depot/main/... //depot/test/..." and making sure that there were no error messages about requiring the -i flag.

$ p4 integ -n //depot/main/... //depot/test/...

//depot/test/jam/fred.txt#1 - integrate from
  //depot/main/jam/fred.txt#4
//depot/test/jam/jam.c#1 - integrate from
  //depot/main/jam/jam.c#2
//depot/test/jam/jam.h#1 - integrate from
  //depot/main/jam/jam.h#2

Resources

The referenced Perl script is available here.

Please send feedback and comments on this page to

Back to the top

Contact us

© Vaccaperna Systems Ltd