rrizun

some 'secret weapon' libraries/technologies

2011-10-25T06:36:00.000-07:00

some 'secret weapon' libraries/technologies

(in no particular order)

libcurl
libevent
libfuse
sqlite3

collectd

corosync (lan)
rabbitmq (wan)

jersey (java)
java.util.concurrent + google guava
jdom

gwt + requestFactory
jquerymobile

mysql
berkeley db je (java edition) (as a java local store)
sqlite3 (as a c/c++ local store)

duplicity (e.g., to amazon s3)
stackoverflow.com
cakephp (fer quick 'n dirty crud style apps)
dreamhost.com

a solid state disk!

eclipse + junit
apt-gettable/yum installable tomcat

boost
CAP theorem

gwt requestFactory

2011-08-19T09:15:00.000-07:00

Quick follow up: turns out gwt requestFactory 'entities' work *great* for cmd/event style

key is to *not* provide request context methods such as persist(PersonProxy p), update(PersonProxy p) and delete(PersonProxy p)

instead, I'm doing something slightly different... e.g,. given:

interface PersonProxy extends EntityProxy {

 ...

 String getEmail();

 String getEmployer();

 ...

}

instead of a 'crud' style interface:

interface PersonService {

Request[Void] update(PersonProxy person);

}

have a cmd(/event) style interface instead:

interface PersonService {

Request[Void] setEmail(PersonProxy person, String email);

Request[Void] setEmployer(PersonProxy person, String employer);

}

a number of reasons/ramifications for doing this:

1. in my server side domain, I only have a cmd style interface; if the server side service signature is update(Person person) then then problem is I don't know which properties changed and therefore I don't know which properties I need to send domain cmds for to change (this is the crux of crud)

if you're just insert/upsert into, say, a sql database (and you don't have no business logic sideeffects) then crud can work perfectly well

this is not the case in my server side domain model

2. furthermore, note that there are no setters in PersonProxy; this is desirable and consistent with cmd/event style -> dtos are read only blobs of data, and data is changed by (a) issuing cmds and (b) receiving events indicating changes to data (from you or not from you) and integrating them back into the dto yourself (cmd/event -> cqrs)

3. the server side is easy to implement; I just keep a local map of PersonId -> Person of persons i've 'seen', and gwt requestFactory findEntity() simply does a lookup from this map

it is my 'list' style service methods (e.g., listAllPeople(int start, int limit)) that actually really populate (on demand) the server side person map from actual data (e.g., maybe a sql select query) anytime and everytime a client request a list of data (perhaps from a CellTable dataProvider)

so, effectivly, the server side person map is a true replicated copy of the actual domain persons (replicated and updated via domain events) -> again, consistent w/cmd/event style arch

furthermore, the server side gwt rf infrastructure always graciously returns the same server side person entity (using my supplied findEntity() method) every time a person service method is invoked, so, implementing server side service methods is easy, e.g.,:

public class PersonServiceImpl {

 ...

 public void setEmail(Person person, String email) {



 // STEP 1 update my local replicated domain

 person.setEmail(email);



 // STEP 2 try to update the actual real domain!

 // e.g., may sql insert, maybe cmd.send on a service bus (I'm making this up:)

 someServiceBus.send("person_channel", new Cmd("setEmail", "email="+email));



}

...

}

Note, here, the entire Person entity is being passed in... the key here is that the client side is logically sending the whole entity (but physically, only the *delta*, which, for all intents and purposes should only be the bits for the EntityProxyId, yes?) but it is really doing so only for identification purposes (I wonder if my service methods signatures could just use EntityProxyId instead then?!? will hafta think about that one)

when setEmail returns, the server side gwt rf servlet infrastructure 'notices' that the service side entity has changed (been updated) and that triggers the server side to 'fire' a WriteOperation.UPDATE event that will be sent back to the client (piggy-backed on the request return) and that will trigger the client side eventBus to fire a EntityProxyChange(Event)

all this happens by virtue of the fact that Person is in the service method's signature

now, here's where it gets really interesting:

now we can really start writing some interesting server side service methods w/interesting business logic side effects that the client side can remain ignorant of, yet still see/discover and act upon those sideeffects

e.g.,


interface PersonService {

...

public Request[Void] createFiveRandomPeople();

...

};

the server side service method implementation is easy... just create the five people and (a) stash 'em in the local person map and (b) try to actually create the five people in the actual domain model (e.g., sql insert or send cmd); this is not the interesting part

the interesting part is, here, this is a piece of service side business logic that the client can invoke, but, the client doesn't really know what it really does, i.e, what are the sideeffects?

surprisingly easy solution: change the method signature:


interface PersonService {

...

public Request[Set[PersonProxy]] createFiveRandomPeople();

...

}

so, now that the newly create people are in the signature of the service method, this will trigger the server side gwt rf servlet to do 'find's for every instance of Person in the signature (input or output parameters?) and this will trigger UPDATE events to be sent back to the client in the response payload which will trigger UPDATE events on the client side

(Note- not quite sure I'm explaining the exactly 100% correctly... I *think* the client side gwt rf infrastructure keeps a set of 'seen' EntityProxyIds and if the client side rf 'sees' a new entity that it hasn't seen before the it locally fires a UPDATE event) (if that's not the way it really works then I don't understand how it works yet =))

back on the client side, say the client has a CellTable of PeopleProxy... the data provider can simply register for EntityProxyChange(Event)s on PersonProxy and can simple do a brute force refresh of the list data... at first glance that might sound inefficient, i.e., why not just specifically add the 5 returns personProxies to the dataProviders list? well, the answer is because the client probably doesn't know *how* to add them business logic wise, i.e., at the beginning? at the end? in the middle? by last name? I dunno... (the server side does though! I guess you could re-code all of the biz logic in the client side, but, that might be impracticable and/or impossible) so, by just doing a brute force refresh of the data provider data then the client side automatically leverages the server side business logic and everything falls in place nicely

phew!

Its funny how the WriteOperation.PERSIST, UPDATE and DELETE make it *appear* that gwt requestFactory is all about crud, and, it sorta is and it sorta isn't

PERSIST is all about crud... in a non-crud application, it ideally should not be relevant

UPDATE and DELETE actually really are not all about crud... they are more about *events*... that is, UPDATE really means, "hey! I just saw this new entity that I hadn't seen before!" (and that doesn't necessarily mean that it was just created, all it means is that you just now just all of a sudden discovered it!)

similarly, DELETE means "Hey! I just saw this entity 'go away' that I previously knew of!"... and that doesn't mean that the client just deleted it! (maybe another client somewhere else deleted it, and you're just 'hearing' about it via a server side event that got lucky enough to be propagated to you because you just happened to refresh a cellTable with a range of data that just happened to have that entity in it previously)

so, in a cmd/event style app, one should expect *not* to use PERSIST (Note- I reserve the right to be slightly wrong about the last statement)

let me elaborate: in a gwt rf app, you'll only ever see PERSIST events if the *client* creates an entire, e.g., PersonProxy and beams it the the server side for 'putting' (i.e., the C in CRUD), that is, you have a service method like Request[Void] putPerson(PersonProxy person);

an alternative style is to have a service method like Request[Void] newPerson(String name, String email, String employer)... now, when the client side formulates these 3 parameters and beams them to the server, the server can 'put' the person (or whatever) and, here's the interesting part, *return* it back to the client; this will trigger the client to fire a UPDATE event (because it hasn't 'seen' the returned person entity proxy before)... notice no "PERSIST" event! (Note- I haven't tried this myself yet, need to do so ... [update 8/20/11] just tried this and it all worked perfectly and exactly as described!) so, the client side service signature is really Request[PersonProxy] newPerson(String name, String email, String employer); and this is much less crud style and much more cmd/event/cqrs style.

and really, this is no different than the "createFiveRandomPeople" scenario... newPerson() is simply just a service method of which the client side has absolutely no clue about what its sideeffects are (i.e., it does not have first class knowledge that a person is gonna be created!) instead, it implicitly learns of the sideffects (lets be honest, can be arbitrary sideeffects, really!) via local client side events that the gwt requestFactory automatically 'detects' by virtue of the entity proxy being in the return part of the method signature

if u really want to be philosophical, in a cmd/event style arch, there are only ever "UPDATE" events =) i.e., the entity is either there or it isn't... you have no control over the entity's existence, so, its either, "UPDATE! I just saw this entity that I didn't see before" or "UPDATE! I no longer see this entity that I saw before" =) i.e., DELETE is just a special kind of UPDATE (Note- again, I reserve the right to be slightly wrong about the last statement)

one 'downside' of this whole cmd/event style w/gwt requestFactory is that it kinda throws a monkey wrench into the gwt Editors framework (but then again, the Editors framework is a lot about crud, yes? so it should not be that surprising- some may consider it a downside, others an upside)

you end up with the same problem on the client side as you would on the server side with a server side service method such as void update(Person p)

on the client side, the Editors framework gives you back a whole PersonProxy... now what? what properties were changed? sure, you could keep deltas, but, its still crud... e.g., *why* were the properties changed? change of address? fix typo? easier to provide such semantics w/cmd/event (imho) I can see using Editors against ValueProxies, but not necessarily against EntityProxies (because then you're fully into the world of crud (which, again, can be fine if it meets/suits requirements))

having said all this, I think the gwt requestFactory framework is wonderful and is already working great in two (2) of my cmd/event style apps

I think it took a little bit longer than expected to ramp up on gwt rf and all of its ramifications but is definitely worthwhile as it will (a) save a lot of grunt work writing boiler plate code and (b) somewhat enforces good client/server style wrt cmd/event style

good stuff

p.s.- just though of an interesting gwt rf idiom: "rename"

imagine

Request[PersonProxy] rename(PersonProxy person, String newName);

pretty slick: on return from server side service impl, request factory will (a) 'notice' that the incoming person is no longer "isLive" and will have a DELETE event fired against it and (b) 'notice' the outgoing Person and do a findEntity on it and 'discover' that it now some how magically and all of a sudden *exists* !! this results in a UPDATE event fired against the 'new' person entity... pretty slick (I hope I'm not lying here...! [update 8/20/11] I'm not lying! just tried this myself and everything worked perfectly exactly as I 'theorized'. except for one subtle server side service method impl detail: the rename implementation should make a deep copy of the server side entity and return that; that triggers the expected DELETE/UPDATE event pair (interesting, if u don't do that then u observe a pair of UPDATE/UPDATE events... hmmmm...))

its nice how everything comes together nicely

using gwt request factory as a generic rpc mechanism

2011-08-14T06:15:00.000-07:00

Been looking at GWT requestFactory lately, but from a different perspective.

My perspective is not one of able to rapidly code a 'crud' style application.

Rather my perspective is one of able to rapidly code a 'cmd/event' style application.

What I'm gonna do is decline all of the GWT requestFactory 'entity' stuff and only use the 'value' stuff (GWT ValueProxy introduced in GWT 2.1.1 I think) to come up with a cmd/event-style application (because in my case, my server side model is a cmd/event style model)

With crud style apps you can have fast up front results at the possible cost of long term unmaintainable code (which can be fine for simple apps).

With cmd/event style apps you can have long term maintainable code at the cost of up front design (complex apps).

(I say "can" because, as with any "tool", any developer can use or mis-use it; cmd/event -> pay me now (w/up front design); crud -> pay me later (w/future maintenance))

Using GWT RPC

GWT RPC apps typically start out something like this:

the developer carefully writes a MyServiceAsync/MyService/MyServiceServlet trifecta and must manually ensure (yes, gpe helps) that every single photon is perfectly aligned =) furthermore, design-wise it is effectively a direct tunnel from client side to server side (may or may not be desirable... I hope not desirable -> tight coupling, but in the grand scheme of things probably not that big of a deal because this client/server interface should really be a 'private' client/service interface anyway whose only purpose is to facilitate communication from the client browser to server servlet).

as gwt rpc apps get more complicated they might morph to something like this:

myClientSideModel might be introduced to reduce tight coupling between client/server

myClientSideModel might be introduced to allow for a client-side event/service bus; the client side can publish local events and the client side ui views can react to those local events and update themselves- pretty slick.

myServiceSideModel might be introduced to allow for a more flexible server side architecture; e.g., it might not be practical/possible to jam everything into a custom server side servlet (e.g., Objectify is independent of servlets); also, might want to unit test server side code, and that might be a little bit more difficult if everything is jammed in a monolithic fashion into a custom server side servlet

Using GWT requestFactory

with gwt requestFactory:

It is pretty much the same as the previous gwt rpc design except that we're missing the boilerplate MyService and MyServiceServlet (and the *Async is replaced with *Request, both playing the same role, but with burdensome restrictions lifted).

Here, MyServiceSideModel is any service-side java code -> not saddled w/being a servlet.

Furthermore, the *Request interface is not tightly coupled at the photon level to a server side component -> as a matter of fact it is not tightly coupled to anything -> it is loosely ducked typed to a server side pojo (non-servlet) component (yet the gwtc compiler give you a hight degree of compile time type-safety, i.e., gwtc verifies at compile time that the duck typing has any hope of succeeding).

As a matter of fact, all of the 'model' code that you write, in both client and server, is just plain java code -> no more boilerplate (well, ok, less boilerplate; developer still writes ValueProxies (a good thing though -> keeps 'em honest) and a RequestFactory (too easy to mention)).

GWT requestFactory advantages wrt GWT RPC for cmd/event style apps:

* rf runs faster in browser because it is json based (browser uses native code to parse json instead of translated java->javascript code to parse gwt rpc payloads)
* rf keeps developer more 'honest' because they need to come up with formal client side value interfaces
* client-server interface is dynamically duck-typed (vs statically typed in gwt rpc)
* less boiler plate code to write (ok, actually, different boiler plate code to write... actually, developer writes duck-typed client side interfaces; one may or may not call that boilerplate code)
* other stuff

this was really just my brain dump of the benefits of using GWT requestFactory instead of GWT RPC as a general RPC mechism; using GWT RF in this way effectively encourages good client/service cmd/event architecture without burdening the developer with boring boiler plate code (that's what all of the GWT RF generators do now) -> and as a bonus it runs fast too.

hope some of this makes sense!

references

http://tbroyer.posterous.com/gwt-211-requestfactory

http://stackoverflow.com/questions/4119867/when-should-i-use-requestfactory-vs-gwt-rpc/4926459#4926459

http://www.udidahan.com/2009/12/09/clarified-cqrs/

ubuntu/wireshark as non-root

2011-07-06T07:14:00.001-07:00

1a. dpkg-reconfigure wireshark-common
1b. say "yes"
2. add a user to wireshark group (e.g., sudo vi /etc/group)
3. logout/login for changes to take effect

using duplicity linux in a nutshell

2011-05-02T08:28:00.000-07:00

duplicity script to backup a user defined set of folders:

e.g.,

duplicity --no-encryption --include /etc --include /home --include /cvsroot --include /cvsrootx --include /workspace --exclude '**' / file:///mnt/backup

be sure the --exclude '**' is last!!

Automatically generate AirPrint Avahi service files for CUPS printers

2011-02-08T06:06:00.000-08:00

https://github.com/tjfontaine/airprint-generate

passwordless ubuntu

2011-01-07T08:14:00.000-08:00

here is how to disable passwords prompts for, e.g., the user "ubuntu":

passwd -d ubuntu
visudo: %admin ALL=(ALL) NOPASSWD: ALL
add user ubuntu to "admin" group
in /etc/pam.d/common-auth: change nullok_secure to just nullok

pjsip/pjsua w/sound on ubuntu 10.10

2010-12-15T11:44:00.001-08:00

sound doesn't seem to work unless pjsua sees the alsa header files at make time

just do a "apt-get install libasound2-dev" and then (re)make the pjsua binary

takeaways on cap theorem

2010-12-13T09:32:00.000-08:00

my rough cryptic notes on cap theorem

cp sacrifices availability
cp -> money, i.e., for stuff that is money related
cp -> for online operation; realtime; returnval

think synchronous request/response, e.g., an atm money machine should probably not spew money if there is no (network connectivity) availability; also, it tells u right then and there ok/error

ap sacrifices consistency
ap -> not money, i.e., not for stuff that is money related
ap -> for offline operation; not realtime; no returnval

think asynchronous queue/pub/sub, e.g., your email client (e.g., outlook, thunderbird) is always 'available' to receive (fire-and-forget) requests to send email

also, cp -> quorum

also, should not bother to try to build ca systems (because p is inevitable)

good to know

2009-11-20T07:28:00.001-08:00

good to know:

>>> There's a bug in JDK 1.6.0_14 which causes breakpoints not to work.
Check which JDK version you have. 1.6.0_16 is out and allegedly fixes
this bug.

http://gosnoozemail.com

2009-04-08T18:30:00.000-07:00

and another friend just launched a few days ago-

http://gosnoozemail.com

http://truthrally.blogspot.com

2009-04-08T18:27:00.000-07:00

FYI a friend of mine is blogging about a bootstrapped startup in california-

http://truthrally.blogspot.com

mysql paging without start limit

2009-04-08T18:04:00.000-07:00

here is a way to iterate thru a large (possibly changing) dataset without using start/limit-

http://www.mail-archive.com/google-appengine@googlegroups.com/msg05973.html

(the same technique can be applied to mysql)

you can iterate a large (possibly changing) dataset using the above "last" method

futhermore (a) the above method can be efficient with large datasets with appropriate indicies and (b) only the last result needs to be remembered

you cannot iterate a changing dataset using a naive "start/limit" implementation

futhermore (a) there are cases where start/limit can be inefficient with large datasets even with appropriate indicies and (b) all results need to be remembered to guard for correctness

bottom line: "start/limit" is the easy/unsafe way to iterate while the above "last" method is the hard/safe way to iterate

Droid Sans Mono

2008-07-11T08:21:00.000-07:00

http://damieng.com/blog/2007/11/14/droid-sans-mono-great-coding-font

fedora+java6+gwt+debug=annoying

2008-05-31T07:55:00.000-07:00

Problem

Sick and tired of the Eclipse debugger breaking here when debugging GWT projects using Java6 on Fedora?

--- snip ---
Exception in thread "Timer-0" java.lang.AbstractMethodError: org.apache.xerces.dom.DocumentImpl.getXmlStandalone()Z
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.setDocumentInfo(DOM2TO.java:373)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:127)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:94)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(TransformerImpl.java:662)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:708)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
at java.util.prefs.XmlSupport.writeDoc(XmlSupport.java:254)
at java.util.prefs.XmlSupport.exportMap(XmlSupport.java:333)
at java.util.prefs.FileSystemPreferences$8.run(FileSystemPreferences.java:607)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.prefs.FileSystemPreferences.writeBackCache(FileSystemPreferences.java:600)
at java.util.prefs.FileSystemPreferences.syncSpiPrivileged(FileSystemPreferences.java:784)
at java.util.prefs.FileSystemPreferences.access$2300(FileSystemPreferences.java:33)
at java.util.prefs.FileSystemPreferences$13.run(FileSystemPreferences.java:754)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.prefs.FileSystemPreferences.syncSpi(FileSystemPreferences.java:752)
at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1317)
at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322)
at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322)
at java.util.prefs.AbstractPreferences.sync2(AbstractPreferences.java:1322)
at java.util.prefs.AbstractPreferences.sync(AbstractPreferences.java:1308)
at java.util.prefs.FileSystemPreferences.sync(FileSystemPreferences.java:731)
at java.util.prefs.FileSystemPreferences.flush(FileSystemPreferences.java:807)
at java.util.prefs.FileSystemPreferences.syncWorld(FileSystemPreferences.java:451)
at java.util.prefs.FileSystemPreferences.access$1200(FileSystemPreferences.java:33)
at java.util.prefs.FileSystemPreferences$4.run(FileSystemPreferences.java:419)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
--- snip ---

Solution

In Eclipse, go to Window -> Preferences -> Java -> Debug and *uncheck* "Suspend execution on uncaught exceptions"

GWT Install Problem on Fedora

2008-05-31T07:51:00.000-07:00

Problem

--- snip ---
** Unable to load Mozilla for hosted mode **
java.lang.UnsatisfiedLinkError: /usr/local/gwt-linux-1.4.62/mozilla-1.7.12/libxpcom.so: libstdc++.so.5: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1751)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1647)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1005)
at com.google.gwt.dev.shell.moz.MozillaInstall.load(MozillaInstall.java:190)
at com.google.gwt.dev.BootStrapPlatform.go(BootStrapPlatform.java:40)
at com.google.gwt.dev.GWTShell.main(GWTShell.java:318)
--- snip ---

Solution

yum install compat-libstdc++-33

change your amazon developer forums nickname!

2008-05-08T17:41:00.000-07:00

FYI You can change your amazon developer forums nickname here:
http://s1.amazon.com/exec/varzea/account-access/settings/104-1775788-0244713?account-seller=1

s3fs r152

2008-05-08T16:52:00.000-07:00

* r152 (May 8, 2008)
+ uses x-amz-copy-source
+ less uploading, rsync works faster!
+ by popular demand s3fs now sets the "correct" Content-type!
+ when creating files, s3fs uses /etc/mime.types to set the "correct" mime type based on file extension!
+ supports symlinks!
+ rsync works better!

UNIXODBC tidbit...

2008-04-09T20:53:00.000-07:00

UNIXODBC tidbit...

In UNIXODBC there are three (3) configurable timeout options:

* login_timeout
* connection_timeout
* query_timeout

LOGIN_TIMEOUT in the UNIXODBC world is MYSQL_OPT_CONNECT_TIMEOUT in the Mysql world.

CONNECTION_TIMEOUT in the UNIXODBC world is MYSQL_OPT_READ_TIMEOUT and MYSQL_OPT_WRITE_TIMEOUT in the Mysql world.

QUERY_TIMEOUT in the UNIXODBC world has to do with canceling a query if the time it takes to execute the query exceeds the timeout.

Here is the kicker: CONNECTION_TIMEOUT is *not* implemented in MYSQL Connector/ODBC, i.e., CONNECTION_TIMEOUT is *not* mapped to MYSQL_OPT_READ_TIMEOUT and/or MYSQL_OPT_WRITE_TIMEOUT. I believe it is mapped to nothing!

FYI s3fs r130

2008-04-01T17:43:00.000-07:00

FYI s3fs r130

* some logging to /var/log/messages (e.g., try "tail -f /var/log/messages")
* set file/folder ownership to whatever getuid()/getgid(0 returns
* fixed that nasty local file cache bug; cp, rsync and friends should now work fine w/local file cache enabled!

building s3fs is a wee little bit more mac-friendly

2008-02-24T13:06:00.000-08:00

building s3fs is a wee little bit more mac-friendly

the steps for building on mac are something like this:

* install Xcode
* install MacFUSE
* install macports
* sudo port install curl
* sudo port install libxml2
* sudo port install openssl
* sudo port install subversion
* svn checkout
* export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
* make!

s3fs: parallel-ized readdir()

2008-02-23T07:38:00.000-08:00

directory listings in s3fs are now a lot faster: I've parallel-ized getattr requests in readdir()

new release of s3fs! now supports rsync!

2008-02-17T21:33:00.000-08:00

New release of s3fs! Now supports rsync by virtue of "x-amz-meta-mode" and "x-amz-meta-mtime" custom meta headers!

Download it at http://code.google.com/p/s3fs svn 43

new amazon sqs pricing

2008-02-06T15:08:00.000-08:00

$0.01 per 10,000 requests

so a worker polling SQS once per second would rack up $2.60 per month in SQS fees... not too bad!

No luck with s3fs and lsb-sdk...

2008-01-22T14:19:00.000-08:00

I tried compiling s3fs via lsb-sdk (linux standard base SDK) on Fedora FC5... it compiled, linked and ran, sorta. I could get directories via a mounted s3 bucket, however, when attempting to, say, "cat" a text file, s3fs would segfault because read() was given a null "fi" parameter...

Compiling s3fs on FC5 *outside* of lsb-sdk and s3fs ran flawlessly. Furthermore, copying the s3fs binary that was built under lsb-sdk on FC5 to a Fedora 7 system: when attempting to run that s3fs binary on Fedora7 gave a "cannot find "libcurl.so.3". Not exactly sure what the "LSB" way of resolving this is...

Suggestions welcome!

installing lsb-sdk clobbers /root

2008-01-21T16:02:00.000-08:00

This is fedora5... Incredibly, running lsb-sdk's install.sh ends up clobbering the ownership / mode of "/root" to this:

drwxrwxr-x 6 1001 users 4096 Apr 3 2007 root

This was near-fatal for me on an Amazon EC2 instance (with ssh public/private key authentication) as it did this without telling me and I was no longer able to ssh to the instance... sshd was "suddenly" complaining about the ownership/mode of "/root":

After I was able to restore ssh to my ec2 instance (don't ask) I noticed this gem in /var/log/secure:

Jan 21 18:00:00 domU-xx-xx-xx-xx-xx-xx sshd[9693]: Authentication refused: bad ownership or modes for directory /root

So, just a warning when installing lsb-sdk, best to restore /root with a "chmod 750 /root" and a "chown root:root /root". Ironic how lsb-sdk of all things does this! =)

Update: did *not* notice this behavior on fedora7 though...

linux standard base

2008-01-16T09:19:00.000-08:00

Got s3fs compiling in linux standard base.

* one trick was to use LSBCC_SHAREDLIBS for libfuse, libcurl and libxml
* another trick was to use -Wl,--allow-shlib-undefined

Hopefully I'll be able to post a binary that runs on most linux distros...!

Reference: http://www.entrepreneur.com/tradejournals/article/133183822_1.html

vncviewer and Amazon EC2

2007-12-17T13:51:00.000-08:00

vncviewer works a heck of a lot better on Amazon EC2 instances compared to remote Xwindows. Here's how I got it running:

yum install vncserver
yum groupinstall "X Window System"

As well, you'll need to use ec2-authorize to open up port 5901.

Once that's done (a) run "vncserver" on your Amazon EC2 instance and (b) run vncviewer on a client!

s3fs

2007-12-15T14:47:00.000-08:00

FYI I've posted a new version of s3fs (svn version 28). It now uses a local file cache (~/.s3fs) for increased performance. As well it, it now retries on 500 server errors, making it a wee little bit more robust.

These changes also lay down the foundation for supporting modified-time and chmod (e.g., tar and rsync).

http://code.google.com/p/s3fs

Google Android=J2ME tutorials

2007-11-26T11:37:00.000-08:00

Is it just me or has there been a flurry of J2ME tutorials ever since Google Android was announced?

Xwindows on Amazon EC2

2007-11-07T09:11:00.000-08:00

I setup Xwindows on an amazon ec2 instance today...

I started with (a) amazon's fc4 image, (b) yum upgrade to fc5, (c) yum groupinstall "GNOME Desktop Environment". That whole process takes about an hour or so.

Then, from my local desktop I tried "ssh -X my.amazon.instance gnome-terminal" but that did not work. Using ssh -v showed that there was no remote xauth (remote on the ec2 instance, that is).

So, on the ec2 instance, "yum install xorg-x11-xauth" did the trick. I was able to start gnome-terminal on a remote ec2 instance and view it on my local desktop.

Performance was bad though: pretty much unusable, probably due to network latency.

ssh-installkeys

2007-10-27T10:23:00.001-07:00

~~very handy: http://www.catb.org/~esr/ssh-installkeys~~

obsolete: use /usr/bin/ssh-copy-id

s3fs rsync and mtime

2007-10-14T08:39:00.000-07:00

Thinking about how to support rsync with s3fs, e.g., changing modification timestamps, etc...

You can provide custom headers (x-amz) at upload time but I see no way to change headers without re-uploading the entire object.

My thought is to go a "proprietary" way: include an additional s3 object per s3 object to hold name/value pairs.

E.g., If I have the object "/mybucket/f7-i386.iso" then there could be another objected called "/mybucket/f7-i386.iso#meta" whose contents can be a simple list of name/value pairs, e.g., "mtime=2005-12-31 23:59:59.9" or something like that.

s3fs would know to look for and interpret/manage those somewhat-non-critical #meta files.

s3fs

2007-10-14T07:38:00.000-07:00

FYI I've posted "s3fs" at http://code.google.com/p/s3fs, a FUSE based file system backed by Amazon S3. It stores files "natively" in S3 (and as such has the same 5GB per file limit) so that you can access the same files with other S3 clients.