Hook program for gitannex to use flickr as backend.
This allows storing any type of file on flickr, not only images and movies.
Requirements:
python2
Credit for the flickr api interface goes to: http://stuvel.eu/flickrapi
Credit for the png library goes to: https://github.com/drj11/pypng
Credit for the png tEXt patch goes to: https://code.google.com/p/pypng/issues/detail?id=65
Install
Clone the git repository in your home folder.
git clone git://github.com/TobiasTheViking/flickrannex.git
This should make a ~/flickrannex folder
Setup
Run the program once to set it up.
cd ~/flickrannex; python2 flickrannex.py
After the setup has finished, it will print the git-annex configure lines.
Configuring git-annex
git config annex.flickr-hook '/usr/bin/python2 ~/flickrannex/flickrannex.py'
git annex initremote flickr type=hook hooktype=flickr encryption=shared
git annex describe flickr "the flickr library"
Notes
Unencrypted mode
The photo name on flickr is currently the key used by git-annex.
Encrypted mode
The current version base64 encodes all the data, which results in ~35% larger filesize.
I might look into yyenc instead. I'm not sure if it will work in the tEXt field.
-- Tobias
https://github.com/TobiasTheViking/flickrannex/issues/3
9 days ago: [the annex] "hook format a few versions ago, and this is using the new hook format".
Looks very handy. I am just starting with this, but can't seem to get it working as a remote after following the simple walkthrough. All goes well until:
my Ubuntu 12.04:
I guess my "git-annex version is still too old"? Any idea what version is needed? Even better if I can figure out which Linux distribution/release has the most up to date version of annex.
Get the statically linked version from here http://git-annex.branchable.com/install/Linux_standalone/
I believe the new hook format was introduced in version 4.20130521
Many thanks.
I used gitannex-install and was left with a slight anomaly:
But I guess this includes the new hook format. I get a bit further:
The path for the binary "/usr/bin/python2" is wrong.
It could be any of /usr/bin/python /usr/bin/python2.6 /usr/bin/python2.7
Or maybe in /usr/local/bin
you can try running "which python" or "which python2" to get the real path.
Thanks, but on my machine I get:
I have scripted all my walkthrough commands, blowing away the test repositories and flickr settings first each time. This re-runs the flickr scripts and git config annex.flickr-hook etc.
I can't spot anything here.
That's weird...
You could try adding "--dbglevel 1 --stderr" arguments to the hook command and give me the output. But the way i read the log it seems like it doesn't even launch the python intrepreter. I might be wrong though.
Thanks and sorry to trouble you, it is my error, I picked unencrypted option (thinking it would be less of an issue) and am using a text file for test, gave an error line:
I've not looked through your code yet, but could that message be printed when not in debug mode?
Hi! Does this backend support chunksize option? If yes, is it possible to set it after the remote has been added to the repository? Thanks, Milan.
I'm not even sure if chunksize is exposed to the hooks at all.
As it is, the hook will check the filesize, and if the filesize is more than 30mbyte it will exit 1.
Chunking may be implemented down the road. I do believe joeyh might have some plans that will touch this issue, so I'd rather wait. Than re-invent the wheel yet again.
Hi, I am coming back to this and testing Flickr as a repository for moving files about and have run into what may be my very basic misunderstanding with vanilla annex.
I copied one file to Flickr and dropped it elsewhere (--force). I assumed that the file was on Flickr ok but that the numcopies setting required the force because of the semi-trust level of the Flickr remote.
Then I find I can't get the file back, even though there is a record of it from whereis.
Can you help enlighten me as to what am I missing? I assumed whereis would only report files that exist and can be copied back. If not my error, I can raise bug or search for logs. Thanks in advance for any help.
[[!format perl """
nrb@nrb-ThinkPad-T61:~/tmp$ git annex whereis whereis libpeerconnection.log (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok whereis test.cgi (1 copy) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) ok whereis walkthrough.sh (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok whereis walkthrough.sh~ (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok nrb@nrb-ThinkPad-T61:~/tmp$ git annex get test.cgi get test.cgi (from flickr...)
git-annex: /home/nrb/tmp/.git/annex/tmp/SHA256E-s48--a01eedbee949120aeda41e566f9ae8faef1c2bacaa6d7bb8e45050fb8df6d09d.cgi: rename: does not exist (No such file or directory) failed git-annex: get: 1 failed nrb@nrb-ThinkPad-T61:~/tmp$
"""]]
Another try - this time a slightly simpler setup using my version of the walkthrough commands
[[!format bash """
nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex drop walkthrough.sh --from usbdrive drop usbdrive walkthrough.sh ok (Recording state in git...) nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex move walkthrough.sh --to flickr move walkthrough.sh (gpg) (checking flickr...) (to flickr...) /home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: /home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: ok (Recording state in git...) nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis whereis walkthrough.sh (1 copy) 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) ok whereis walkthrough.sh~ (3 copies) 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) 7803d853-d231-4bb4-b696-f12a950fb96b -- here (my laptop) d60d75f9-d878-4214-af20-fa055134ae77 -- usbdrive (portable USB drive) ok nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex get walkthrough.sh get walkthrough.sh (from flickr...) (gpg) git-annex: /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--02f600d7e8b071d2945270fd5e7fc26dd066ff31: openBinaryFile: does not exist (No such file or directory) gpg: decrypt_message failed: eof
Unable to access these remotes: flickr
Try making some of these repositories available: 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) failed git-annex: get: 1 failed nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr fsck walkthrough.sh (gpg) (checking flickr...) (fixing location log) * Based on the location log, walkthrough.sh * was expected to be present, but its content is missing.
* No known copies exist of walkthrough.sh failed fsck walkthrough.sh~ (checking flickr...) (fixing location log) Based on the location log, walkthrough.sh~ * was expected to be present, but its content is missing. failed (Recording state in git...) git-annex: fsck: 2 failed nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$
""" ]]