C.J. Adams-Collier (cjcollier) wrote,
C.J. Adams-Collier

Description of MaxDB code tree

I had a conversation with Daniel Dittmar, one of the developers on the SAP team today. I am trying to get familiar with the MaxDB codebase, and it's difficult to do without at least a bit of explanation. The inline English documentation is written by native German speakers, so it's often difficult to discern. Having this map will help me quite a bit. I hope it helps some others as well.

Thanks very much to Daniel for putting an evening in to documenting this.

12:56 < cj> DanielD: can you point me to some docs on the maxdb source code?
12:56 < cj> why are the different parts named with such short and
difficult-to-remember names? :)
12:57 < DanielD> partly, because if you work with them day in, day out, you
remember them anyway and they are easier to type
12:57 < cj> :)
12:58 < DanielD> what documentation there is, is either in comments (doxygen
format) or in sys/doc on the cvs server
12:58 < cj> is there any documentation on the structure of the code?
12:59 < cj> ah. I'd like to generate some html from the doxygen, is this
12:59 < DanielD> but the latter is mostly in german, out of date and incomplete
12:59 < DanielD> yes, but I'm not sure it helps much
12:59 < DanielD> I can give you a quick rundown of the directories
12:59 < cj> I'll take anything I can get :)
12:59 < cj> please do
13:05 -!- CGI582 [n=43b7eadb@] has joined #maxdb
13:05 < CGI582> hey, look. web-based irc.
13:05 < CGI582> http://colliertech.org/cgi-bin/cgiirc/irc.cgi
13:05 -!- CGI582 is now known as cj_
13:16 < Raimi> yeah, cgiirc is nice
13:18 -!- DanielD [n=danieldi@n114-224.dsl.de.inter.net] has quit [Read error:
113 (No route to host)]
13:24 < cj> for some value of "nice" :)
13:24 < cj> I still prefer screen + irssi, but that's just because I am a
beautiful and unique snowflake.
13:25 < cj> http://tirania.org/blog/archive/2005/Nov-27-1.html
13:29 -!- DanielD [n=danieldi@tc17-n68-202.de.inter.net] has joined #maxdb
13:31 < DanielD> cj: sorry for the interuption
13:32 < DanielD> I could give you a rundown of the directories if you're
13:38 < cj> please do. That would be a great help.
13:38 < DanielD> ok
13:39 < DanielD> ak and SAPDB/SQLManager is the SQL layer
13:39 < cj> can we start with ak and bd?
13:39 < cj> :)
13:40 < DanielD> I thought I try to bridge the old style and the new style
13:40 < DanielD> but come to think of it, I'm not sure I get it right every time
13:40 < cj> sounds good. What's the difference between old style and new style?
13:40 < DanielD> old style was mainly pascal and the very short file names
13:41 < DanielD> new style is C++ and DirectoriesWithVeryLongDescriptiveNames
13:41 < DanielD> ak is SQL layer
13:41 < DanielD> bd is B* trees (table data to pages, indices etc)
13:42 < DanielD> bw: ignore, something with BusinessWarehouse, but it's not
linked to the kernel unless I'm mistaken
13:42 < DanielD> cn is part of the database manager (the old version was called
13:43 < DanielD> co stands for COM == liveCache or Object Management System,
the first version was built using MS COM
13:44 < DanielD> config is for build configuration, not database configuration
13:44 < DanielD> ContentStorage: not quite sute, belonged to the SAP Content
13:45 < DanielD> en = environment: the Unix runtime
13:45 < DanielD> eo = en + os: parts of the runtime that are common to Windows
and Unix
13:46 < DanielD> extern: librraries not written by us (there are some more, but
we started to put them into one place)
13:46 < DanielD> gg: code shared by the old kernel layers (ak, kb, bd)
13:47 < DanielD> Globals: a description containing compile options for all
modules, sort of like global CFLAGS
13:48 < DanielD> idl: contained tools to translate COM idl files to some xml
used on Unix (we were using MS MIDL on Windows)
13:49 < DanielD> in: contained code common to all clients, not used anymore
except a few routines used by the Loader for date/time
13:49 < DanielD> install: the MaxDb installer
13:50 < DanielD> Interfaces: only the program to register the ODBC driver
13:50 < DanielD> ip: something from the precompiler, not sure why it needs it's
own directory
13:51 < DanielD> jv: the JDBC driver. It's actually developed in another tree,
but we put the driver here so that we can put it inbto the
13:51 < DanielD> kb: Kernel Logging and Locking
13:52 < DanielD> lctools: some scripts to register a liveCache
13:52 < DanielD> libptc: the Pascal runtime library
13:53 < DanielD> livecache: some tests for liveCache
13:53 < DanielD> but mostly obsolete
13:53 < DanielD> ls: the MaxDb Loader
13:53 < DanielD> msvc70: a lib to write backtraces on Windows
13:54 < DanielD> mt: was tracing for the old clients
13:55 < DanielD> ni: interfacing to backup tools (do not confuse with NI: the
SAP Networtk Interface, which is also mentioned somewhere)
13:55 < DanielD> os: Windows runtime environment
13:56 < DanielD> ox: I have no idea, so it can't be important
13:56 < DanielD> pa: C precompiler
13:57 < Raimi> some of the acronyms do not make sense :)
13:58 < DanielD> That's because they were german:ak = Auftrag Kommunication =
Order communication
13:58 < Raimi> pa ?!
13:59 < DanielD> no idea. p is precompiler,
13:59 < DanielD> possibly pa = Precompiler Auftrag
13:59 < Raimi> scary
13:59 < Raimi> possible
13:59 < DanielD> because ak, kb, bd are really like a domino game
14:00 < DanielD> Auftrag to Kommunikation (Kommunikation was the communication
between distributed databases)
14:00 < DanielD> kb: Kommunikation to Basis
14:01 < DanielD> bd: Bassis to Daten
14:02 < DanielD> pi, pr: also precompiler: some parts are still used by the
ODBC driver as well, but that's currently being rewritten on
top of SQLDBC
14:03 < DanielD> ptoc: the header file required to translate the ptoc generated
C files: explanation of this directory would take another
evening because actually two files get merged
14:04 < DanielD> px: also ODBC, but only one file, so I ignore it
14:04 < DanielD> r3: some scripts used during installation of R/3
14:05 < DanielD> redist: the python interpreter used during loading of the
system tablees
14:05 < DanielD> Resource: Windows ressource file so you see something when you
do right click/Properties/Version
14:05 < cj> do you mind me posting this on my blog?
14:06 < DanielD> not at all. I did a version some years ago, but it's kinda out
of date
14:07 < cj> great
14:07 < DanielD> rx: terminal definition files: there's only of client left
that uses these and it's used only for support purposes, so we
ignore it
14:08 < DanielD> sap: some header file where we interface to SAP libraries: the
mentioned Network Interface and a library so that SAP app
servers can talk to the database manager via SAP's Remote
Funtion Call
14:08 < DanielD> SAPDB: the new world
14:09 < DanielD> sp: routines used by clients and server alike
14:10 < DanielD> contains the conversions of our number format (vsp4* and
vsp5*), unicode conversion and 'stuff'
14:10 < DanielD> ta: the tracing routines of the kernel
14:11 < DanielD> ut: a diagnose tool for support (this is the remainder of the
very first administration tool, how the mighty have fallen)
14:12 < DanielD> wd: the web agent: used to implement Web DBm and WebSQL
14:12 < DanielD> xt: belongs also to the support tool in ut: ut was the
official part and xt was the secret support stuff
14:13 < DanielD> xx: the scripts used to load the system tables
14:13 < cj> *whew*
14:14 < Raimi> so much code and still it scanes indices only with 500k/s :)
14:14 < DanielD> ok, next mall
14:14 < DanielD> Raimi: noch so 'ne freche Bemerkung und Du fliegst raus, Milz
14:14 < Raimi> ja, otto
14:14 < DanielD> SAPDB:
14:15 < DanielD> ConsistentRead: implementation of Multi Version Read (not
functional yet)
14:16 < DanielD> Container: our replacement for C++ STL (the Open Source STL
didn't work very well with custom allocators, which is
essential for us)
14:17 < DanielD> Converter: Kernel: maps logical page numbers to volumes and
physical page numbers
14:17 < cj> d'oh. I would have liked to see the Open Source STL cleaned up
instead of re-inventing it. Oh well...
14:18 < DanielD> CppSQL: the sql interface used to write C++ stored procedures
14:19 < DanielD> DataAccess: Kernel: layout of data pages and B* trees
14:19 < DanielD> DBanalyzer: Client: performance monitoring
14:20 < DanielD> DBM: Client: database manager
14:20 < DanielD> DBProc: Server: Java Stored Procedures: not complete yet
14:21 < DanielD> FileDirectory: Server: roughly equivalent to inodes on Unix
14:22 < DanielD> management of tables and indices, but the mapping of table
names to these entries is somewhere else (in the SQL catalog)
14:23 < DanielD> FrameControl: Server: something releate to DataAccess, but I'm
too tired to fake any knowledge
14:24 < DanielD> FreeBlockmanagement: memory menagement of the data cache
(which caches data pages in memory)
14:24 < DanielD> GarbageCollector: removes deleted liveCache object from data
14:25 < DanielD> Installutil: Client: part of the installer
14:25 < DanielD> Interfaces: SQLDBC, new ODBC, PHP, DBD::MaxDB, native
communication for JDBC
14:26 < DanielD> IOManager: Server: sits between the data cache and the runtime
(which does the actual I/O)
14:27 < DanielD> Join: Server: implementation of SQL Joins
14:27 < DanielD> KernelCommon: obvious
14:28 < DanielD> liveCache: obvious (I like obvious, I should use it more often)
14:28 < DanielD> Logging: Server: obvious
14:28 < Raimi> so IOManager is the one that needs the speed-up ?
14:28 < DanielD> LVCSimulator: AFAIK a debugging environment for developers of
liveCache stored procedures
14:30 < DanielD> There's currently work being done so that adjacent logical
pages become also physical adjacent pages to speed IO
14:30 < DanielD> and I guess thats part of the IOManager
14:30 < DanielD> MemoryDebugger: also part of the debugging environment for
liveCache stored procedures
14:31 < DanielD> Message: a framework to pass lots of error information to
upper levels
14:31 < DanielD> OMS: Kernel: Object Management System = liveCache
14:32 < DanielD> sorry, I'm switching between 'Kernel' and 'Server', but both
mean the same
14:33 < DanielD> PacketInterface: C++ interface to our communication structure
between clients and Kernel
14:33 < DanielD> Pager: belong also somewhere between DataAccess and IOManager
14:34 < DanielD> Purify: = rational Purify, a memory debuggerm which
unfortunately never really worked with the kernel
14:34 < DanielD> Registry: cross platform configuration store
14:35 < DanielD> RegistryUpdate: why does every
14:35 < DanielD> why does everyone has to create new directories?
14:36 < DanielD> Restart: restarting the database, especially applying the log
after an abort
14:37 < Raimi> that's all really interesting
14:37 < Raimi> i'll read it in cj's blog
14:37 < Raimi> sleep well :)
14:37 -!- Raimi [i=raimi@port-195-158-171-223.dynamic.qsc.de] has quit ["Client
14:38 < DanielD> RunTime: the operating system specific parts
14:40 < DanielD> Samples: some examples for the interfaces
14:40 < DanielD> (Pager and Restart are also Kernel)
14:40 < DanielD> SAPDBCommon: code shared by clients and Kernel
14:41 < DanielD> Scripting: Python driver and Perl interfaces to dbm and loader
14:42 < DanielD> ServerTasks: Kernel: coordination of the tasks that are
executed in parallel (like backup and creating of indices)
14:42 < DanielD> SQLManager: Kernel: the SQL layer
14:43 < DanielD> SQLStudio: WebSQL
14:44 < DanielD> syncman: client: the Synchronization manager: only java
binaries, the code is developed in another tree
14:44 < DanielD> SystemViews: Kernel: SQL access to Kernel internal data,
counter and statistics
14:45 < DanielD> Table: Kernel: the objects that are handled by FileDirectory,
so it's a bit more low level than an SQl table
14:45 < DanielD> ToolsCommon: Client: code shared by the tools (dbm, loader,
sqlstudio etc.)
14:46 < DanielD> Tracve: kernel:
14:46 < DanielD> Trace: Kernel: database trace
14:47 < DanielD> Transaction: Kernel: don't know exactly which part of
transactions are handled here and which are handled in Logging
14:48 < DanielD> WebAgent: Framework used by WebDBm and WebSQl
14:48 < DanielD> WebDAV: WebDAV server which stores everything in MaxDB
14:49 < DanielD> XML: some beginnings of XML indexing
14:49 < DanielD> This concludes our lesson for today
14:49 < DanielD> there will be a little quiz about this next week
Tags: c, c++, irc, maxdb, pascal, sap, work

  • Post a new comment


    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.