Results published by Google
Google has posted a summary of the results of the Summer of Code on the Company Blog. There’s also a link in there to a cool map with all the participants and mentors.
Posts tagged 'samba' – Page 4
scons issues
While I was very optimistic about scons a month ago, I must admit that enthusiasm has faded a bit now that we tried out to use it for building Samba4. The idea behind it is good, but there are some issues when using it in real life (not all of them huge surprises):
- It assumes the relations between subsystems are compositional. If dir a is processed before dir b, dir a can not depend on any object files from dir b. This is a major problem for Samba.
- Running the configure tests (even with caching) each time you run a build takes way too much time for a project with as much tests as Samba (althought there are hacks around it).
- No easy way to add configure test functions.
- Python. This is a problem for Samba both because it means an extra dependency and not a lot of committers are familiar with it.
- Slow. Calculating checksums for each file rather then relying on modification times may be nice for those that have their sources on NFS, but it slows scons down a lot.
- Very much focussed on files and conversions between files - generating a list of init functions like we do at the moment is quite hard.
- While in theory it should be possible to generate a configure script and Makefile.in from a set of SConscript files, this proved to be a bit harder then I’d hoped.
Of course, it’s not all bad. Some good things about scons:
- Can transparently handle portability issues with different compilers, architectures
- The support for multiple build types is very neat and useful
- Automatic support for header dependencies
- Easy to extend. Scanning for dependencies, adding file types, adding compilers is not hard.
- Compositional to a large degree. Being able to put the list of files and the list of functions/headers a subsystem needs in one file in that subsystems dir is very useful.
I’m not really sure where to go to from here. I am no longer convinced scons is the right way to go, though using some of the ideas from scons in the current build system might be good.
Pidl and compatibility #2
Apparently MIDL does support something that is equivalent subcontext() in pidl. It has two attributes, represent_as, and transmit_as, which do something similar and are just about what I need for subcontext()s replacement. Looks like the only thing left to figure out now is what the arguments to the conversion functions will be.
Pidl and compatibility
One of the areas of pidl I am currently to improve is compatibily with other IDL compilers, so that exchanging IDL files becomes easier. We currently have extensions in pidl (with good reason!), that are unfortunately not supported by other compilers. My aim is therefore to not make support for these extensions a requirement for being able to load IDL files that use them. The easiest way to support something like that is by removing a few of the pidl-specific attributes before compiling with another compiler.
A recent change I made to pidl was the addition of support for the [string] attribute that is traditionally used in DCE/RPC IDL to make strings (which are nothing more then zero-terminated arrays, as far as NDR is concerned). For example, a string can be declared as follows in traditional IDL:
1 | [string] wchar *bla; |
pidl previously had it’s own way of handling strings (a “string” type with special flags to indicate whether it was in ASCII, unicode, fixed-length, etc) which had the advantage that it would take care of handling the conversion between whatever character encoding was used on the wire and whatever one was used by Samba internally. The string specified above would look something like this:
1 | [flag(STR_SIZE4|STR_LEN4)] string *bla; |
In order to support [string] and also do on-the-fly character set conversions, I also added a new [charset()] attribute that tells Samba from what character set to convert. The string above would now look something like this:
1 | [string,charset(UTF16)] wchar *bla; |
One of the other extensions that needs to be a bit more optional is the [subcontext] attribute. This attribute is currently used for interpreting various generic data blobs that are send using DCE/RPC and traditionally interpreted by another layer in Windows. Subcontexts allow us to merge these two layers in Samba. For example the following code on windows:
1 2 | uint32 size; [size_is(size)] uint8 data[]; |
(data would then by casted to a FOO_BAR on a higher level), would look like this in Samba at the moment:
1 2 | uint32 size; [subcontext(4)] FOO_BAR data; |
The problem here is that [subcontext()] is not very flexible. It’s argument is not very intuitive to use, and there are two other attributes available that change its behaviour.
I’d like to, in the future, replace subcontext with something like this:
1 2 | uint32 size; [size_is(size),replace(FOO_BAR)] uint8 data[]; |
Pidl would then call the push / pull / print functions of FOO_BAR after processing data[]. The uint8 array would only be declared locally in the pull / push / print function and “FOO_BAR data;” would appear in the public structures.
I’m not yet clear on what the semantics of the pull / print / push functions for FOO_BAR would be. Requiring names like ndr_{pull,push,print}_FOOBAR and the standard NDR arguments would certainly make sense, but it would also be great if we could support custom non-NDR subcontexts more easily (such as TDR-generated or hand-written ones). A possible solution would be allowing something like [replace(FOO_BAR,tdr)], maybe having the prefix default to “ndr”?
pidl-generated code now in Samba3
The first autogenerated RPC interface (DFS) is now in SVN, thanks to Jeremy. Hopefully the rest will follow soon.
Samba3 backend to pidl
Two days ago, I finally took the time to sit down and attempt to write a Samba3 backend to pidl (the IDL compiler in Samba 4). Turned out to be much easier then I expected! It took me two evenings (~16 hours total) to write basic NDR parser, header, server, client and template generators.
The code currently generated runs inside of the Samba3 tree without modifications (uses no additional helper functions and the like). I’m currently focussing on making the DFS interface autogenerated. Most things seem to work now, except for some name changes (the functions in Samba3 and Samba4’s IDL don’t have the same names). I’m working on that now. The latest generated sources are available at http://samba.org/~jelmer/pidl_samba3/ .
Pidl now in CPAN
Encouraged by the fact that ethereal now uses pidl as part of its’ standard build, I tried to upload Pidl as a package to CPAN today. Turned out to be much easier to become a perl developer then to become a Debian developer… :-)
More information is available at Pidl at CPAN.
SoC Deadline
So, just a few more minutes until the deadline of the Google Summer of Code, 2005! I got pretty much all that I wanted in there done, except for the ldb_map module, which is still a bit dodgy.
Participating was a nice and interesting experience! The other participants that were working on Samba for SoC have produced some great code and it looks like they’re going to stick around. Especially Brads’ work on DSSYNC looks promising. I can certainly recommend anyone to participate if Google is repeating SoC again sometime in the future!
Looking back, I guess it could’ve been interesting to work on a different project then Samba such as mono (I do actually actively follow mono development, I just don’t contribute at the moment), althought what I did now was probably more productive and allowed me to spend some time on other sides of Samba than I am usually involved in as well.
I just committed the file STATUS file to the tree:
—- Samba3 -> Samba4 Upgrade —- (C) 2005 Jelmer Vernooij <jelmer@samba.org> Published under the GNU GPL
Sponsored by the Google Summer of Code program (http://code.google.com/summerofcode.html) Mentored by Andrew Bartlett <abartlet@samba.org> Thanks!
- Done:
- Reading wins.dat
- Reading registry.tdb
- Reading passdb.tdb
- Reading account_policy.tdb
- Reading group_mappings.tdb
- Reading winbindd_idmap.tdb
- Reading share_info.tdb
- Reading secrets.tdb
- Reading smbpasswd
- Reading + writing (generic) smb.conf files
- Testsuite for read support mentioned above
- Console utility for dumping Samba information
- Import user accounts in Samba4
- Import groups in Samba4
- Import secrets in Samba4
- Import WINS data in Samba4
- Dump idmap data to LDB
- Import registry keys/values in Samba4
- Import account policies in Samba4
- Testsuite for upgrade
- Console utility from upgrading from Samba3 -> Samba4
- SWAT (Web interface) support for upgrading from Samba3 -> Samba4
- LDB generic mapping module
- (Experimental) Samba4 LDB <-> Samba3 LDAP mapping module based on LDB generic mapping module
- Testsuite for Samba4 LDB <-> Samba3 LDAP mapping module
Source files: source/lib/ldb/modules/ldb_map.c source/lib/ldb/modules/ldb_map.h source/lib/samba3/group.c source/lib/samba3/idmap.c source/lib/samba3/policy.c source/lib/samba3/registry.c source/lib/samba3/samba3.c source/lib/samba3/secrets.c source/lib/samba3/share_info.c source/lib/samba3/smbpasswd.c source/lib/samba3/tdbsam.c source/lib/samba3/winsdb.c source/lib/samba3/samba3.h source/scripting/libjs/upgrade.js source/scripting/ejs/smbcalls_param.c source/scripting/ejs/smbcalls_samba3.c source/param/generic.c source/param/generic.h testdata/samba3/verify testprogs/ejs/samba3sam source/setup/upgrade source/scripting/bin/samba3dump source/dsdb/samdb/ldb_modules/samba3sam.c source/script/tests/test_s3upgrade.sh swat/install/samba3.esp
- Known remaining issues:
- [upgrade] Conversion from the smbpasswd/TDB passwords to ntPwdHash / lmPwdHash is broken. Couldn’t find out why.
- [ldb_map] Conversion of attribute names in DN’s is still a bit dodgy
- [ldb_map] mapped objectClass names may be mentioned multiple times in returned records
- [ldb_map] add/modify support not tested very well with LDAP yet (only LDB+TDB)
- [ldb_map] group membership is not yet mapped (only primaryGroupID / sambaPrimaryGroupSID)
The alternative approach
Looking more closely at one of the other approaches I could’ve taken for migrating from Samba3. We will probably end up supporting this method later on anyway, once we expand the vampire code. I do think the path I’ve taken for SoC is the right one though - it allows the upgrade with the least hassle and user input.
This approach could be splitted up into the various parts:
- samba3 pidl backend
- Samba4 vampire + server side samsync support in Samba3
- unixinfo (unixinfo) (exposes idmap) - in Samba4 (client side) - in Samba3 (server side)
- winsrepl (thru seperate pipe?)
- enum/add shares (srvsvc)
- enum/add registry (winreg)
- enum/add printers (spoolss(?))
- convert smb.conf (using Jerry’s registry hack)
Advantages of this approach:
- Can be used to ‘clone’ Samba4 boxes as well
- Relatively few extra loc necessary (a lot shared with vampire)
- ‘Clean’ code - simply enumerates and adds
Disadvantages:
- Needs running Samba3 daemon for upgrade (!)
- Needs (complex?) Samba3 code changes
- Needs latest version of Samba3
Partially working upgrade paths
The one-way upgrade from Samba3 to Samba4 is now partially working! There are both a web-frontend and a console utility that can do an upgrade, given a libdir (usually /var/lib/samba) and a configuration file (usually in /etc/samba/smb.conf).
Last night, I rewrote most of the conversion code to be in EJS rather then in C. One of the positive things about this is that I can use embedded LDIF inside of the JavaScript code and that I can build on the existing Samba4 provisioning code (which is in EJS as well). Writing EJS wrappers for my existing C code was easier then I had thought.
One of the other things I did today is add a fallback mechanism for ldb_map where it can store the data that could not be mapped. There are now two types of records in ldb_map: those that are partially mapped (mapped, but might have one or two attributes stored in the fallback database) and those that are completely stored in the fallback database.
Below are some screenshots of the upgrade procedure in SWAT: /blog/uploads/screenshots/upgr1.png /blog/uploads/screenshots/upgr2.png /blog/uploads/screenshots/upgr3.png
LDIF output working
The sam.ldif and winbind.ldif generation code now works as well. I’ll be working on generating the new smb.conf next.
One of the things that worries me is the fact that I can generate so few of the Samba4 ‘user’ and ‘group’ attributes. Probably need to borrow some code from the current provisioning script to fix that up.
I’m wondering whether I should do some of the upgrade using EJS, so it can be integrated into SWAT easily…
ldb_map somewhat working
ldb_map and the samba3sam ldb module that is based on it work nicely now and appear to be memory-leak free. I have so far been testing with a TDB backend loaded with an LDIF from Samba3. Tomorrow will be the big day, testing against an OpenLDAP server.
Since the LDAP schema for Samba3 contains information that the Samba4 SAM database doesn’t store and vice versa, it will be interesting to find out whether the current mapping will be sufficient. If it is not, I might have to fall back to storing the Samba4 attributes I can’t store in the Samba3 LDAP database in a seperate database instead.
Another thing I did today is clean up the parameter list in Samba4 - I removed all unused parameters. This should hopefully make it easier to do the Samba3->Samba4 smb.conf conversion code.
TODO is down to:
- smb.conf conversion
- Finish sam.ldif / winbind.ldif from the TDB files
- Test samba3sam module against OpenLDAP
First LDIF working
Today was a productive day. I just got the LDIF output for samba3dump working for wins.ldif and the various registry hives. TODO is down to:
- sam.ldif
- winbind.ldif
- smb.conf
- test and finish ldb_map and ldb_samba3
I hope to get the first three done tomorrow.
Samba4 debian packages
Now that the Samba4 Debian packages appear to be somewhat working, I’ve set up a daily cronjob that generates packages for the latest SVN revision. Add the following line to your “/etc/apt/sources.list” to try them out:
1 | deb http://jelmer.vernstok.nl/debian/autobuild . |
Please note that these packages are still very much alpha, as is the Samba4 code. Comments are welcome
Status Update
Althought I haven’t been writing my weekly Samba4 status updates for a while now, development on Samba4 certainly hasn’t been stalled. There have been a lot of improvements since June. I’ll try to sum up most of them here, hopefully not forgetting too much:
- ldb_sqlite is done afaik, still need to try it
- Deryck and Tridge have been working on SWAT, giving it a user-friendly interface
- Lots of progress in the directory synchronisation interface by Brad (Summer of Code participant) and Metze
- Andrew has done more work on Kerberos integration such as the kpassword protocol and the PAC
- Yet more LDB updates from Simo, not sure what exactly :-)
- Quite a couple of things have been converted to Javascript. ejs is now being used for some of the testing as well as the provisioning script. Mostly by Tridge and Mimir
- pidl updates and build system updates from me
… and much, much, more
I’m personally still a bit skeptical on using ejs for things other then SWAT, but I can’t really describe why. Perhaps it’s the fact that when I used one of its early incarnations, Javascript was very hairy to use. I also don’t get that “warm fuzzy feeling” when I use Javascript that I get when I use C, C#, Haskell or Python, but I guess that’s not really a good argument. :-) The C interfaces to EJS are certainly very easy to use and EJS is a nice small library.