Thanks to the help of Steinar Gunderson in the past week, the Debian packages of Samba4 have had some major improvements. I have once again started to provide snapshots. As usual, they can be used by adding the following line to your /etc/apt/sources.list:
1 | deb ftp://jelmer.vernstok.nl/pub/debian ./ |
Google has posted a summary of the results of the Summer of Code on the Company Blog. There’s also a link in there to a cool map with all the participants and mentors.
While I was very optimistic about scons a month ago, I must admit that enthusiasm has faded a bit now that we tried out to use it for building Samba4. The idea behind it is good, but there are some issues when using it in real life (not all of them huge surprises):
Of course, it’s not all bad. Some good things about scons:
I’m not really sure where to go to from here. I am no longer convinced scons is the right way to go, though using some of the ideas from scons in the current build system might be good.
Apparently MIDL does support something that is equivalent subcontext() in pidl. It has two attributes, represent_as, and transmit_as, which do something similar and are just about what I need for subcontext()s replacement. Looks like the only thing left to figure out now is what the arguments to the conversion functions will be.
One of the areas of pidl I am currently to improve is compatibily with other IDL compilers, so that exchanging IDL files becomes easier. We currently have extensions in pidl (with good reason!), that are unfortunately not supported by other compilers. My aim is therefore to not make support for these extensions a requirement for being able to load IDL files that use them. The easiest way to support something like that is by removing a few of the pidl-specific attributes before compiling with another compiler.
A recent change I made to pidl was the addition of support for the [string] attribute that is traditionally used in DCE/RPC IDL to make strings (which are nothing more then zero-terminated arrays, as far as NDR is concerned). For example, a string can be declared as follows in traditional IDL:
1 | [string] wchar *bla; |
pidl previously had it’s own way of handling strings (a “string” type with special flags to indicate whether it was in ASCII, unicode, fixed-length, etc) which had the advantage that it would take care of handling the conversion between whatever character encoding was used on the wire and whatever one was used by Samba internally. The string specified above would look something like this:
1 | [flag(STR_SIZE4|STR_LEN4)] string *bla; |
In order to support [string] and also do on-the-fly character set conversions, I also added a new [charset()] attribute that tells Samba from what character set to convert. The string above would now look something like this:
1 | [string,charset(UTF16)] wchar *bla; |
One of the other extensions that needs to be a bit more optional is the [subcontext] attribute. This attribute is currently used for interpreting various generic data blobs that are send using DCE/RPC and traditionally interpreted by another layer in Windows. Subcontexts allow us to merge these two layers in Samba. For example the following code on windows:
1 2 | uint32 size; [size_is(size)] uint8 data[]; |
(data would then by casted to a FOO_BAR on a higher level), would look like this in Samba at the moment:
1 2 | uint32 size; [subcontext(4)] FOO_BAR data; |
The problem here is that [subcontext()] is not very flexible. It’s argument is not very intuitive to use, and there are two other attributes available that change its behaviour.
I’d like to, in the future, replace subcontext with something like this:
1 2 | uint32 size; [size_is(size),replace(FOO_BAR)] uint8 data[]; |
Pidl would then call the push / pull / print functions of FOO_BAR after processing data[]. The uint8 array would only be declared locally in the pull / push / print function and “FOO_BAR data;” would appear in the public structures.
I’m not yet clear on what the semantics of the pull / print / push functions for FOO_BAR would be. Requiring names like ndr_{pull,push,print}_FOOBAR and the standard NDR arguments would certainly make sense, but it would also be great if we could support custom non-NDR subcontexts more easily (such as TDR-generated or hand-written ones). A possible solution would be allowing something like [replace(FOO_BAR,tdr)], maybe having the prefix default to “ndr”?
The first autogenerated RPC interface (DFS) is now in SVN, thanks to Jeremy. Hopefully the rest will follow soon.
Two days ago, I finally took the time to sit down and attempt to write a Samba3 backend to pidl (the IDL compiler in Samba 4). Turned out to be much easier then I expected! It took me two evenings (~16 hours total) to write basic NDR parser, header, server, client and template generators.
The code currently generated runs inside of the Samba3 tree without modifications (uses no additional helper functions and the like). I’m currently focussing on making the DFS interface autogenerated. Most things seem to work now, except for some name changes (the functions in Samba3 and Samba4’s IDL don’t have the same names). I’m working on that now. The latest generated sources are available at http://samba.org/~jelmer/pidl_samba3/ .
Encouraged by the fact that ethereal now uses pidl as part of its’ standard build, I tried to upload Pidl as a package to CPAN today. Turned out to be much easier to become a perl developer then to become a Debian developer… :-)
More information is available at Pidl at CPAN.
So, just a few more minutes until the deadline of the Google Summer of Code, 2005! I got pretty much all that I wanted in there done, except for the ldb_map module, which is still a bit dodgy.
Participating was a nice and interesting experience! The other participants that were working on Samba for SoC have produced some great code and it looks like they’re going to stick around. Especially Brads’ work on DSSYNC looks promising. I can certainly recommend anyone to participate if Google is repeating SoC again sometime in the future!
Looking back, I guess it could’ve been interesting to work on a different project then Samba such as mono (I do actually actively follow mono development, I just don’t contribute at the moment), althought what I did now was probably more productive and allowed me to spend some time on other sides of Samba than I am usually involved in as well.
I just committed the file STATUS file to the tree:
—- Samba3 -> Samba4 Upgrade —- (C) 2005 Jelmer Vernooij <jelmer@samba.org> Published under the GNU GPL
Sponsored by the Google Summer of Code program (http://code.google.com/summerofcode.html) Mentored by Andrew Bartlett <abartlet@samba.org> Thanks!
Source files: source/lib/ldb/modules/ldb_map.c source/lib/ldb/modules/ldb_map.h source/lib/samba3/group.c source/lib/samba3/idmap.c source/lib/samba3/policy.c source/lib/samba3/registry.c source/lib/samba3/samba3.c source/lib/samba3/secrets.c source/lib/samba3/share_info.c source/lib/samba3/smbpasswd.c source/lib/samba3/tdbsam.c source/lib/samba3/winsdb.c source/lib/samba3/samba3.h source/scripting/libjs/upgrade.js source/scripting/ejs/smbcalls_param.c source/scripting/ejs/smbcalls_samba3.c source/param/generic.c source/param/generic.h testdata/samba3/verify testprogs/ejs/samba3sam source/setup/upgrade source/scripting/bin/samba3dump source/dsdb/samdb/ldb_modules/samba3sam.c source/script/tests/test_s3upgrade.sh swat/install/samba3.esp
Looking more closely at one of the other approaches I could’ve taken for migrating from Samba3. We will probably end up supporting this method later on anyway, once we expand the vampire code. I do think the path I’ve taken for SoC is the right one though - it allows the upgrade with the least hassle and user input.
This approach could be splitted up into the various parts:
Advantages of this approach:
Disadvantages:
The sam.ldif and winbind.ldif generation code now works as well. I’ll be working on generating the new smb.conf next.
One of the things that worries me is the fact that I can generate so few of the Samba4 ‘user’ and ‘group’ attributes. Probably need to borrow some code from the current provisioning script to fix that up.
I’m wondering whether I should do some of the upgrade using EJS, so it can be integrated into SWAT easily…
ldb_map and the samba3sam ldb module that is based on it work nicely now and appear to be memory-leak free. I have so far been testing with a TDB backend loaded with an LDIF from Samba3. Tomorrow will be the big day, testing against an OpenLDAP server.
Since the LDAP schema for Samba3 contains information that the Samba4 SAM database doesn’t store and vice versa, it will be interesting to find out whether the current mapping will be sufficient. If it is not, I might have to fall back to storing the Samba4 attributes I can’t store in the Samba3 LDAP database in a seperate database instead.
Another thing I did today is clean up the parameter list in Samba4 - I removed all unused parameters. This should hopefully make it easier to do the Samba3->Samba4 smb.conf conversion code.
TODO is down to:
Today was a productive day. I just got the LDIF output for samba3dump working for wins.ldif and the various registry hives. TODO is down to:
I hope to get the first three done tomorrow.
Next on my list is the generation of the LDIF’s for the one-way upgrade path. These will be generated by internally building ldb_ldif structs and then dumping them using the LDB helper functions. This would easily allow us to later on directly write to LDB database.
I’ll need to generate the following LDIF files:
I’ll also make a start on the smb.conf output.
Finally got the ldb_map code done. This is a simple skeleton module that takes a mapping table of LDAP attributes and objectclasses and then maps between two different schema’s. Also got the initial part of ldb_samba3 compiling, which is the first module that actually uses ldb_map.
Yesterday evening, I also added support for reading smb.conf files and share_info.tdb files to the lib/samba3/ directory. Looks like my TODO-list is now down to:
