-
Notifications
You must be signed in to change notification settings - Fork 50
Chroma FAQ
A1: Typically you can clone software modules like chroma with the command:
git clone --recursive git@github.com:JeffersonLab/chroma.git
This should create a working directory called chroma in your current directory. This project directory contains a fully fledged clone of the directory on the github.com server, with all the history etc.
Chroma is built over QDP++ and also QMP for builds targeted to run over some parallel architecture, such as MPI. So, you will also want
git clone --recursive git@github.com:usqcd-software/qdpxx.git
git clone --recursive git@github.com:usqcd-software/qmp.git
The links for QDP and QMP have information about those packages.
A2: Yes. It is in /usr/bin/git on both qcd12kmi and qcdgpu. Be aware that other versions may exist.
A3: You should clone the repository as normal:
git clone --recursive git@github.com:JeffersonLab/chroma.git
cd chroma
The development branch is called remotes/origin/devel
To switch to this branch, the best way is to create a local branch tracks this development branch. You can do this with the command:
git checkout --track -b devel remotes/origin/devel
Git will create a local branch called devel which tracks the remote branch remotes/origin/devel. It will also switch you over to that branch. You can check this by typing
git branch
And git should respond with:
* devel
master
to indicate you are now on the devel branch.
In this sense, tracking means that when you do
git pull
git should automatically try to pull its updates from the devel branch. The same should be true for git push
Q4: I am working with person X. S/He's made changes in his repository. I can access his/her repository. How do I get his/her changes?
A: The best way to do this is probably to create a branch in your own code, pull the changes into that branch, and perform a merge with your working branch. You can pull into your working branch directly, but then it may be slightly trickier to unwind changes.
If you don't yet have a working directory. You can clone the repo of person X
Let us assume for the sake of argument that person X is user bjoo who has a chroma repository in /home/bjoo/Devel/QCD/jlab-standard-chroma-build/chroma/chroma in his home directory on qcd12kmi and that this directory is readable by members of the lattice group.
Let us also assume that user Y is called fred and that he is also a member of the group lattice on qcd12kmi. Then on qcd12kmi (or qcd7ni02) user fred can access the directory. In particular he can clone from there using
git clone file:///home/bjoo/Devel/QCD/jlab-standard-chroma-build/chroma/chroma_
locally.
Cloning remotely (eg to a laptop) on the same network may need access via SSH. Passwords may or may not be needed depending on whether person Y is using an agent with a suitable key:
git clone ssh://qcd12kmi/home/bjoo/Devel/QCD/jlab-standard-chroma-build/chroma/chroma
For offsite access one may need to set up a SSH tunnel and clone through that
If user Person Y (fred) already has a working directory on his machine, and just wants to pull in changes from Person X (bjoo). That can be done in the following steps:
- Create a 'remote' (remote repository) to pull from
- Pull from the remote, possibly into a new branch for safety
- Merge this new branch with the current working branch
Let us take things in turn:
A 'remote' in git, is basically an alias for a(n) URL to a repository. One can add one of these to an existing repository by using the command
git remote add remote_name remote_url
Typically the source of the first clone is kept as origin. So, for example, to add the repo of person X (bjoo) from the previous example as a remote in his/her own repo, person Y can do
git remote add bjoo-ssh ssh://qcd12kmi/home/bjoo/Devel/QCD/jlab-standard-chroma-build/chroma/chroma
One can see it the addition was successful, by typing git remote
bjoo@bertie:~/chroma$ git remote
bjoo-ssh
origin
one can observe that there is now a remote called bjoo-ssh there. One can get more details by adding the -v flag to the git remote command.
bjoo@bertie:~/chroma$ git remote -v
bjoo-ssh ssh://qcd12kmi/home/bjoo/Devel/QCD/jlab-standard-chroma-build/chroma/chroma (fetch)
bjoo-ssh ssh://qcd12kmi/home/bjoo/Devel/QCD/jlab-standard-chroma-build/chroma/chroma (push)
origin git@github.com:JeffersonLab/chroma.git (fetch)
origin git@github.com:JeffersonLab/chroma.git (push)
Now that we have a remote, we can pull or fetch updates from it. To avoid making mistakes it may be a good idea to pull or fetch onto a different branch. We can create and switch to the new branch with:
git branch fromBalint
git checkout fromBalint
Now we can pull in the changes from the remote
git pull bjoo-ssh
Now we can merge the changes into our own master branch. First we switch back to our master branch (or whichever branch we were working on):
git checkout master
(use the appropriate branchname instead of master here if originally you were not working on the master branch). Now we perform the merge:
git merge fromBalint
This means: merge the branch fromBalint onto my current branch. Assuming that the merge is successful, or that any conflicts are suitably resolved, one can delete the branch that the changes were pulled into. It no longer serves a purpose:
git branch -d fromBalint
NB: One does not need to split off a new branch to pull changes into. One can pull it directly into one's current copy. A pull is a git fetch followed by a git merge. One can execute these steps separately if one is unsure that the merge will go smoothly.
A5: This question is essentially addressed in Q4, with person X being the remote repository on qcd12kmi and person Y being the repository on the laptop
A6: Yes. For example if you want to work in a group of say 3 people, each of the three can add remotes for each others repos to pull from. Alternatively a single repo can be set up to which all 3 have write access.
A7 : To write to a repository you need to have SSH access to the machine, and local file access permissions to the repository area. If this situation is satisfied you should be able to move your changes to the remote repository using the command
git push
However, one should have a care: pushing onto a repository which has a checked out working directory may not update that working directory (unless appropriate hooks are set up) and this can cause confusion. It will look to the pusher that the push succeeded, but this will not be reflected on the state of files for the local user. This situation can be circumvented in two ways:
- If working with a collaborator: push to a new branch. Then the local user can merge those into his working branch
- Push to a bare repository. Most server based repositories should be bare. The repositories on github are bare repositories.
Let us address these in turn:
Assume I am working on my laptop and want to push onto qcd12kmi where my repository is at /home/bjoo/Devel/QCD/jlab-standard-chroma-build/chroma/chroma. On my laptop I am in my master branch and have committed my changes, and my local laptop repository is clean.
If I have not yet, I should set up a remote to the qcd12kmi repo:
git remote add qcd12kmi ssh://qcd12kmi/Devel/QCD/jlab/standard-chroma-build/chroma/chroma
(NB: I assume I can reach qcd12kmi via SSH. From offsite, I need to use a tunnel. Likewise, the path starts from my homedirectory because it is my account) Once the remote is set up I can push to a new branch on it:
git push qcd12kmi master:upstream
This will push my master branch, to a branch called upstream on the qcd12kmi remote. I know there is no upstream branch there, so this should create one.
Then on the qcd12kmi machine I can get the changes by merging the upstream branch with my master.
qcd12kmi$ cd Devel/QCD/jlab-standard-chroma-build/chroma/chroma
qcd12kmi$ git merge upstream
The complexity of the merge depends on how much my laptop diverged from the qcd12kmi repo. It may be a simple case of a fast forward merge - where I have just linearly continued on my laptop from the last commit on qcd12kmi, and there are no conflicts. This is the best situation
If one needs to write to github we can set that up. email bjoo AT jlab.org
You probably would want to write to gihub if you are a code maintainer, e.g.: Someone who wants to use git to maintain a project that was hitherto in CVS...(Hint Hint!) or want to make changes that you want to publish to the big bad world.
If you would like to make a contribution to the github repos - and we hope you do - we suggest you use the fork services on github. If you go to the main Chroma software page, you will see a fork button in the upper write part of the page. You can click that to make your own copy of the repo, but this will be under your account at github. You can then clone the repo. to your machine, make changes, commit those changes to your local repo., and then make a pull request - a button also to the right top of the chroma page. This request will notify the owners of the repo. (Chroma) and present them with a web-page detailing the changes made. The owners can then accept merge them into the repo. Instructions for fork/pull are under the github page
Q8: What is a bare repository? What repository is not bare? Can I make a bare repository out of a not-bare repository?
A-8: A bare repository is the part of your repository that lives in the .git subdirectory of your not-bare repository. Or to put it another way, when you have a repository with a working directory, the repository is really just the .git subdirectory, and the rest of the files are loosely speaking the result of a git checkout. Put yet another way, a bare repository is a repository without its working directory and files around it.
By convention, a bare repository ends in the file extension .git. Hence remote repositories, which are usually bare repositories end in the extension .git for example
git@github.com:usqcd-software/qdpxx.git
For reasons of consistency it is generally not a good idea to push into the checked out branch of a not-bare repository. One would update its .git subdirectory, but not necessarily the working files and this can lead to confusion.
The remote repositories on git@github.com:usqcd-software/qdpxx.git are all bare repositories
To extract a bare repository from a non-bare repository is straightforward. Assume I am in a working repository of chroma. I want to extract the bare repository. I come up out of the chroma directory:
cd ..
so the working directory chroma is now a subdirectory. I can then clone it with the --bare option:
git clone --bare chroma
This should create a directory called chroma.git which can then be cloned, pushed to, pulled from etc as normal.
If you'd like to share a project, especially one where you want others to push to it may be a good idea to make a shared clone of it, and then re-clone that and work from there.
A-9: Typically this error message is the result of trying to do something that is not permitted (e.g. pushing onto a repo via the git:// protocol, or not having write access to a directory during a push with the ssh:// protocol.
Check the remote you are trying to push to. You can use the command:
git remote status -v
If you do not give a remote to git push it will try to push to either origin or, if you are on a tracking branch, it will try to push to whichever remote that branch is tracking.
Currently the git protocol (URLs starting with git://) can only be used for reading, not writing. If you have cloned, using the git protocol you need to add a remote via the SSH protocol (URLs starting with ssh://).
If you are trying to push via the SSH protocol and seem to get halfway through a write and then get an error of the sort:
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 291 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
error: insufficient permission for adding an object to repository database ./objects
fatal: failed to write object
error: unpack failed: unpack-objects abnormal exit
To git@github.com:usqcd-software/qdpxx.git
! [remote rejected] master -> master (n/a (unpacker error))
error: failed to push some refs to ' git@github.com:usqcd-software/qdpxx.git'
this means that you have access to the remote, but some local file (eg: group write) permission is messed up. This should be rare. If this happens when pushing onto github and you have already set up access, and things ought to otherwise be working, please contact bjoo AT jlab.org
A10: A merge may fail due to conflicts. These need to be resolved You will be able to find conflicts in files surrounded by chevrons ( <<<<< and >>>>> symbols) like in CVS, SVN etc. Simply keep the correct branch of code. Once you've done editing the files you'll need to do a
git add
on the modified files, and once all the merges are done, you will need to do a
git commit
or alternatively you may be able to get away with one big
git commit -a
The files which do not conflict get added into the GIT Staging Area, but won't be committed (until your final commit). If you get confused and want to go back to the state prior to starting the merge you should be able to do that with the command
git reset --hard
which should blow away the staging area and all the conflicting files. At this point you can try again...
A11: Submodules are evil things. In this case, they are basically other git projects that need to be pasted into the source code of the current project, for the current project to compile.
In one sense they arise naturally due to the modular nature of scidac software. Each software 'product' is an independent module. To assemble a concrete version, you need to pull these together. In our work we tend to put these sub-modules into the other_libs/ subdirectory of a module.
CVS supported this submodule like behaviour by allowing 'module alieases'. By pasting in the source code of another module into the source of the current module it made it easy to bundle packages so the user would only have to do one checkout and one build. The alternative would have been to individually checkout and build and install all the submodules (QDP++ has 4, one of which is recursive) chroma has also got 4 or so.) We considered this second option more tedious than the module aliasing.
Module aliasing lead to other problems tho. When people broke a submodule, it would break the build of the parent module too. SVN and git both avoid this by recording the version of the submodule (SVN remote) and linking it to the version of the superproject in some way.
A purist may say that we should probably not have any submodules (currently a source code issue) but flatten everything out and install the other little modules up front and just link against them thus making this a build system issue. Until we settle on a build system that makes this easy and comfortable for both the developers and the users alike we probably will stick with the existing setup.
A12: Submodules managed by git are so called 'headless' repos. No one version is checked out. It is not really expected that you should change them. If you need to change a submodule, first of all you have to change them from headless repo to a checked out repo. You can do this with the command (in the submodule directory):
git checkout master
which will check out the master branch of the submodule. You can make edits, commmits and pushes in the submodule directory now. When finished, do a git push to publish the changes.
However, this is not enough, because while we have now changed the submodule, its parent project still has the previous submodule version wired into it. Unlike CVS, (and much like an SVN remote) a parent keeps track of the concrete version of its submodule. To pick up the changes in the parent, we have to go to the parent project and do a git add and a git commit on the subdirectory of the submodule
We start off in the top chroma directory. We go to the submodule:
cd other_libs/cpp_wilson_dslash_
At this point cpp_wilson_dslash is headless. It is at the last version which we recorded in the superproject. Let us get the master branch, pull down any updates, work, commit and push etc:
git checkout master
git pull
where the second line consulted the canonical origin repo for any updates before our own edits. We now edit the file and finish with a commit and a push
git commit -a -m "Updated submodule"
git push
We can now go back up to the superproject (chroma) dir:
cd ../..
We can see that our submodules are now out of sync by looking at the output of git submodule status:
git submodule status
ee28205d665fefbdc36b38174b4c023870baa972 other_libs/cg-dwf (chroma3-30-4)
+048828c508d48bfeed8617c7fe0d803b784da0c4 other_libs/cpp_wilson_dslash (chroma3-36-1-2-g048828c)
3b87aa5c7ab0becb23807b3beb67a7f94bd2edcf other_libs/qdp-lapack (chroma3-36-1-4-g3b87aa5)
bbebb315b3a84c1b78da34ac208028bcd1813abf other_libs/sse_wilson_dslash (chroma3-36-1-1-gbbebb31)
observe the "+" sign in front of the second line corresponding to cpp_wilson_dslash. It means that the source in the submodule subdirectory is ahead of what is recorded in the chroma repo. This should be because, we've made changes to the submodule (inlcuding pulling in updates) etc.
To tell chroma to use the new updated version, we need to
bjoo@bertie:~/chroma$ git add other_libs/cpp_wilson_dslash
bjoo@bertie:~/chroma$ git commit -m "Updated cpp_wilson_dslash"
Now if we type git submodule status we get:
ee28205d665fefbdc36b38174b4c023870baa972 other_libs/cg-dwf (chroma3-30-4)
048828c508d48bfeed8617c7fe0d803b784da0c4 other_libs/cpp_wilson_dslash (chroma3-36-1-2-g048828c)
3b87aa5c7ab0becb23807b3beb67a7f94bd2edcf other_libs/qdp-lapack (chroma3-36-1-4-g3b87aa5)
bbebb315b3a84c1b78da34ac208028bcd1813abf other_libs/sse_wilson_dslash (chroma3-36-1-1-gbbebb31)
and we see that the "+" sign is gone. At this point, if we are committers, we can do a git push and push the changes up to the repo.
A13: Check out the Git home page and the references in their documentation, including their online books like ProGit. There is documentation on github as well.
A14: No! Please don't do this. You'll shoot yourself in the foot. The command git init prepares a non-git directory for usa as a git repository. When you clone from github the cloning process automatically includes an invocation of git init as part of the cloning process. Loosely speaking git clone URL is a compound of git init ; git remote add origin URL ; git pull origin master. So in other words git init has already been doen for you by the clone and a second invocation reinitialize the already initialized git repository. Simply: don't do it. The git clone takes care of it.
Do use git init if you are starting a brand new project in a directory where there is not a git repository in existence.
A15: There are various ways of inspecting your history. You can always type
git status
this is a useful command that can give you a lot of information. For example on a tracking branch it will tell you if you are ahead of the branch which you are tracking. The master branch of a clone is always a tracking branch. So if you make a commit in the master branch and type git status you should get back something like:
git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
nothing to commit (working directory clean)
so you know you've advaced by one commit since your last update (pull) of your master branch. To find out what you've changed since your last update recall that you are tracking a branch called remotes/origin/master (or just origin/master) with your own master branch. When you've done your last pull, the state of the origin/master was recorded locally. You can list the log since that pull with
git log remotes/origin/master..
and only the log entries since your last pull will be shown. The .. notation should really be from..to where from and to are branch tags. You can leave off the to like we do above. That implies that to is the current state.
Note that all your commits are tagged by a SHA key. Your commit log will show these:
bjoo@bertie:~/chroma$ git log origin/devel..devel
commit ae2eb37a5844836ae8fa084c1c17a828495fbdb3
Author: Balint Joo <bjoo@jlab.org>
Date: Mon Nov 30 14:37:18 2009 -0500
Updated cpp_wilson_dslash
The long string of numbers: ae2eb37a5844836ae8fa084c1c17a828495fbdb3 is the commit ID. Unlike CVS and SVN, the commit is not per file, but labels the state of your entire repository. In fact, branchnames, and tags are simply just pointers to commit IDs.
It is worth noting, that git log is very versatile and can produce many interesting output formats. See the appropriate chapter of ProGit for details.
Also, there are many git browsers that let you look at the git repository and its history. One such viewer is called gitk and it comes as part of git. To run it, type
gitk
in your working directory.
A16: Go to the main software page for the repo. For chroma, this is Chroma. On that page, one can see the commit history on any of the branches.
To find changes since your last pull, is a little less easy. First you need to find the commit ID of the commit you've last pulled. Then you can correlate this with the history in the remote repository in several ways. First and foremost find the commit id of when you last pulled:
git log -1 origin/master
(if you are working in the master branch). You'll get something like this:
commit 2d9c6d05d999006a178b6d0f9a52fab6ab40cb30
Author: ...
Date: Wed Nov 25 13:26:50 2009 -0500
added pt colorvec
You can now see what has changed in the remote branch since this date/log message combination by looking online at Chroma
An alternative method is to use the
git fetch
command. If you are on the master branch, this will update the local origin/master branch, but unlike the when using git pull it will not try to merge the origin/master branch onto your master branch. The differences will appear in terms of changed (but not yet committed) files (modified/added) when you check_
git status
you can of course also inspect these with any good local repository browsing tool such as gitweb. If you don't want to merge these changes, you can blow them away with
git reset --hard