README
@@ -2,46 +2,97 @@ Getting started with the Userland Consolidation Building the bits Getting Started This README provides a very brief overview of the gate, how to retrieve a copy, and how to build it. Detailed documentation about the Userland gate can be found in the 'doc' directory. Questions or comments about the gate can be addressed to userland-discuss@opensolaris.org. Overview The Userland consolidation maintains a Mercurial gate at ssh://anon@hg.opensolaris.org//hg/userland/gate This gate contains build recipies, patches, IPS manifests, etc. necessary to download, prep, build, test, package and publish open source software. In order to build the contents of the Userland gate, you need to clone it. Since you are reading this, you probably already have, but in any event you can do so with the following command $ hg clone ssh://anon@hg.opensolaris.org//hg/userland/gate /scratch/clone In order to build the bits either individually or collectively, you must set the WS_TOP environment variable to point to the top of your workspace. $ export WS_TOP=/scratch/clone To build and publish the entire contents of the gate, you can use $ cd /scratch/clone $ gmake publish To build and publish a specific component you need to initialize the workspace by building the tools and creating a repository to publish your results in. The easiest way to do this is to $ cd /scratch/clone $ gmake setup Once you have initialize the the workspace, you can build individual components by $ cd /scratch/clone/components/(component) $ gmake publish All of the bits are are built will be published to the repository created by the setup step (file:///scratch/clone/repo/) If you build the entire contents of the gate, individual build logs for each component will be located at /scratch/clone/logs/(target):(component).log The build infrastructure is similiar to that of the SFW consolidation in that it makes use of herarchical Makefiles which provide dependency and recipe information for building the components. In order to build the contents of the Userland gate, you need to clone it. Since you are reading this, you probably already have. Getting the Bits As mentioned, the gate is stored in a Mercurial repository. In order to build or develop in the gate, you will need to clone it. You can do so with the following command $ hg clone ssh://anon@hg.opensolaris.org//hg/userland/gate /scratch/clone This will create a replica of the various pieces that are checked into the source code management system, but it does not retrieve the community source archives associated with the gate content. To download the community source associated with your cloned workspace, you will need to execute the following: $ export WS_TOP=/scratch/clone $ cd /scratch/clone/components $ gmake download This will use GNU make and the downloading tool in the gate to walk through all of the component directories downloading and validating the community source archives from the gate machine or their canonical source repository. There are two variation to this that you may find interesting. First, you can cause gmake(1) to perform it's work in parallel by adding '-j (jobs)' to the command line. Second, if you are only interested in working on a particular component, you can change directories to that component's directory and use 'gmake download' from that to only get it's source archive. Building the Bits. You can build individual components or the contents of the entire gate. Regardless of how you build the gate, you must set WS_TOP in the calling environment to point to the top of your workspace. Ex: $ export WS_TOP=/scratch/clone Component build If you are only working on a single component, you can just build it using following: setup the workspace for building components $ cd ${WS_TOP}/components ; gmake setup build the individual component $ cd (component-dir) ; gmake publish Complete Top Down build Complete top down builds are also possible by simply running $ cd ${WS_TOP}/components $ gmake package-install The 'package-install' target will build each component, publish it to the workspace IPS repo and install it in the running environment. As a result, it is strongly recommended that you only perform complete top down builds in a zone. Tools to help facilitate build zone creation will be integrated shortly. If the zone you create to build your workspace in does not have networking enabled, you can pre-download any community source archives into your workspace from the global with: $ cd ${WS_TOP}/components $ gmake download You can add parallelism to your builds by adding '-j (jobs)' to your gmake command line arguments. The gate should only incrementally build what it needs to based on what has changed since you last built it. components/Makefile
@@ -39,20 +39,21 @@ include components.mk download: TARGET = download prep: TARGET = prep build: TARGET = build install: TARGET = install publish: TARGET = publish validate: TARGET = validate clean: TARGET = clean clobber: TARGET = clobber download: TARGET = download prep: TARGET = prep build: TARGET = build install: TARGET = install publish: TARGET = publish install-packages: TARGET = install-packages validate: TARGET = validate clean: TARGET = clean clobber: TARGET = clobber prep build install publish: TEMPLATE_ZONE=$(ZONE) prep build install publish: LOG = >$(WS_LOGS)/$@.$(TARGET).log 2>&1 .DEFAULT: publish download prep build install publish validate: $(COMPONENT_DIRS) download prep build install publish install-packages validate: $(COMPONENT_DIRS) COMPONENT_DIRS.nosetup = $(COMPONENT_DIRS:%=%.nosetup) @@ -82,14 +83,14 @@ echo "export PATH WS_TOP" >>$@ $(COMPONENT_DIRS): setup FORCE @cd $@ ; echo "$(TARGET) $@" ; \ @cd $@ && echo "$(TARGET) $@" && \ $(BASS_O_MATIC) $(TEMPLATE_ZONE:%=--template-zone %) \ $(@:%=--component %) --make $(TARGET) $(LOG) $(@:%=--component %) --make $(TARGET) $(LOG) $(COMPONENT_DIRS.nosetup): FORCE @cd $(@:%.nosetup=%) ; echo "$(TARGET) $(@:%.nosetup=%)" ; \ @cd $(@:%.nosetup=%) && echo "$(TARGET) $(@:%.nosetup=%)" && \ $(BASS_O_MATIC) $(TEMPLATE_ZONE:%=--template-zone %) \ $(@:%.nosetup=--component %) --make $(TARGET) $(LOG) $(@:%.nosetup=--component %) --make $(TARGET) $(LOG) # depends.mk is auto-generated by the build tools, bass-o-matic.py in particular depends.mk: components.mk components/bind/Makefile
@@ -25,7 +25,7 @@ COMPONENT_NAME= bind COMPONENT_VERSION= 9.7.0-P2 IPS_PKG_NAME = network/dns/bind COMPONENT_CLASSIFICATION="Applications/Internet utility" COMPONENT_CLASSIFICATION="Applications/Internet" COMPONENT_DESCRIPTION= "Berkeley Internet Name Domain server and utilities" COMPONENT_SRC= $(COMPONENT_NAME)-$(COMPONENT_VERSION) COMPONENT_PROJECT_URL = http://www.isc.org/software/bind components/bind/bind.p5m
@@ -29,6 +29,7 @@ set name=info.source_url value=$(COMPONENT_ARCHIVE_URL) set name=info.upstream_url value=$(COMPONENT_PROJECT_URL) set name=org.opensolaris.consolidation value=$(CONSOLIDATION) set variant.arch=$(ARCH) license COPYRIGHT license="ISC" dir path=etc dir path=usr @@ -490,12 +491,11 @@ link path=usr/lib/liblwres.so.60 target=liblwres.so.60.0.0 link path=usr/sbin/named-compilezone target=named-checkzone link path=usr/share/man/man8/named-compilezone.8 target=named-checkzone.8 depend fmri=pkg:/SUNWcs@0.5.11-0.143 type=require depend fmri=pkg:/library/libxml2@2.7.6-0.143 type=require depend fmri=pkg:/library/security/openssl@0.9.8.14-0.143 type=require depend fmri=pkg:/library/zlib@1.2.3-0.143 type=require depend fmri=pkg:/system/library/math@0.5.11-0.143 type=require depend fmri=pkg:/system/library@0.5.11-0.143 type=require depend fmri=pkg:/library/libxml2 type=require depend fmri=pkg:/library/security/openssl type=require depend fmri=pkg:/library/zlib type=require depend fmri=pkg:/system/library/math type=require depend fmri=pkg:/system/library type=require
@@ -29,6 +29,7 @@ set name=info.source_url value=$(COMPONENT_ARCHIVE_URL) set name=info.upstream_url value=$(COMPONENT_PROJECT_URL) set name=org.opensolaris.consolidation value=$(CONSOLIDATION) set variant.arch=$(ARCH) license COPYING license="BSD-like" dir path=usr dir path=usr/bin @@ -62,8 +63,8 @@ variant.arch=$(ARCH) link path=usr/lib/libmagic.so target=libmagic.so.1.0.0 link path=usr/lib/libmagic.so.1 target=libmagic.so.1.0.0 depend fmri=pkg:/library/zlib@1.2.3-0.143 type=require depend fmri=pkg:/system/library@0.5.11-0.143 type=require depend fmri=pkg:/library/zlib type=require depend fmri=pkg:/system/library type=require
@@ -20,6 +20,7 @@ # # Copyright (c) 2010, Oracle and/or it's affiliates. All rights reserved. # include ../../make-rules/shared-macros.mk COMPONENT_NAME= lcms @@ -131,3 +132,5 @@ REQUIRED_FILE += /usr/lib/$(MACH64)/libtiff.so REQUIRED_FILE += /usr/lib/libjpeg.so REQUIRED_FILE += /usr/lib/libtiff.so include ../../make-rules/depend.mk components/lcms/lcms.p5m
@@ -29,6 +29,7 @@ set name=info.source_url value=$(COMPONENT_ARCHIVE_URL) set name=info.upstream_url value=$(COMPONENT_PROJECT_URL) set name=org.opensolaris.consolidation value=$(CONSOLIDATION) set variant.arch=$(ARCH) license COPYING license="LCMS" dir path=usr dir path=usr/bin @@ -58,10 +59,10 @@ file path=usr/bin/wtpt variant.arch=$(ARCH) file path=usr/include/lcms/icc34.h file path=usr/include/lcms/lcms.h file path=usr/lib/$(MACH64)/liblcms.la variant.arch=$(ARCH) #file path=usr/lib/$(MACH64)/liblcms.la variant.arch=$(ARCH) file path=usr/lib/$(MACH64)/liblcms.so.1.0.19 variant.arch=$(ARCH) file path=usr/lib/$(MACH64)/pkgconfig/lcms.pc variant.arch=$(ARCH) file path=usr/lib/liblcms.la variant.arch=$(ARCH) #file path=usr/lib/liblcms.la variant.arch=$(ARCH) file path=usr/lib/liblcms.so.1.0.19 variant.arch=$(ARCH) file path=usr/lib/pkgconfig/lcms.pc file path=usr/share/man/man1/icc2ps.1 @@ -75,11 +76,11 @@ variant.arch=$(ARCH) link path=usr/lib/liblcms.so target=liblcms.so.1.0.19 link path=usr/lib/liblcms.so.1 target=liblcms.so.1.0.19 depend fmri=pkg:/image/library/libjpeg@0.5.11-0.143 type=require depend fmri=pkg:/image/library/libtiff@0.5.11-0.143 type=require depend fmri=pkg:/library/zlib@1.2.3-0.143 type=require depend fmri=pkg:/system/library/math@0.5.11-0.143 type=require depend fmri=pkg:/system/library@0.5.11-0.143 type=require depend fmri=pkg:/image/library/libjpeg type=require depend fmri=pkg:/image/library/libtiff type=require depend fmri=pkg:/library/zlib type=require depend fmri=pkg:/system/library/math type=require depend fmri=pkg:/system/library type=require
@@ -30,6 +30,7 @@ set name=info.source_url value=$(COMPONENT_ARCHIVE_URL) set name=info.upstream_url value=$(COMPONENT_PROJECT_URL) set name=org.opensolaris.consolidation value=$(CONSOLIDATION) set variant.arch=$(ARCH) license LICENSE license="BSD-Like" dir path=usr dir path=usr/include @@ -63,8 +64,8 @@ variant.arch=$(ARCH) link path=usr/lib/libmng.so target=libmng.so.1.0.0 link path=usr/lib/libmng.so.1 target=libmng.so.1.0.0 depend fmri=pkg:/image/library/libjpeg@0.5.11-0.146 type=require depend fmri=pkg:/library/lcms@1.19-0.146 type=require depend fmri=pkg:/library/zlib@1.2.3-0.146 type=require depend fmri=pkg:/system/library/math@0.5.11-0.146 type=require depend fmri=pkg:/system/library@0.5.11-0.146 type=require depend fmri=pkg:/image/library/libjpeg type=require depend fmri=pkg:/library/lcms type=require depend fmri=pkg:/library/zlib type=require depend fmri=pkg:/system/library/math type=require depend fmri=pkg:/system/library type=require components/pycups/pycups.p5m
@@ -29,6 +29,7 @@ set name=info.source_url value=$(COMPONENT_ARCHIVE_URL) set name=info.upstream_url value=$(COMPONENT_PROJECT_URL) set name=org.opensolaris.consolidation value=$(CONSOLIDATION) set variant.arch=$(ARCH) license COPYING license="GPLv2" dir path=usr dir path=usr/lib @@ -43,9 +44,9 @@ file path=usr/lib/python2.6/vendor-packages/64/cups.so variant.arch=$(ARCH) file path=usr/lib/python2.6/vendor-packages/cups-1.0-py2.6.egg-info file path=usr/lib/python2.6/vendor-packages/cups.so variant.arch=$(ARCH) depend fmri=pkg:/library/print/cups-libs@1.4.2-0.143 type=require depend fmri=pkg:/runtime/python-26@2.6.4-0.143 type=require depend fmri=pkg:/system/library@0.5.11-0.143 type=require depend fmri=pkg:/library/print/cups-libs type=require depend fmri=pkg:/runtime/python-26 type=require depend fmri=pkg:/system/library type=require
@@ -29,6 +29,7 @@ set name=info.source_url value=$(COMPONENT_ARCHIVE_URL) set name=info.upstream_url value=$(COMPONENT_PROJECT_URL) set name=org.opensolaris.consolidation value=$(CONSOLIDATION) set variant.arch=$(ARCH) license COPYING license="GPLv2" dir path=usr dir path=usr/bin @@ -39,7 +40,7 @@ file path=usr/bin/rsync variant.arch=$(ARCH) file path=usr/share/man/man1/rsync.1 file path=usr/share/man/man5/rsyncd.conf.5 depend fmri=pkg:/system/library@0.5.11-0.143 type=require depend fmri=pkg:/system/library type=require
@@ -55,7 +55,7 @@ CONFIGURE_OPTIONS += CXX="$(CCC)" CONFIGURE_OPTIONS.32 += --bindir=/usr/bin CONFIGURE_OPTIONS.32 += --libdir=/usr/lib CONFIGURE_OPTIONS.64 = --bindir=/usr/bin/$(MACH64) CONFIGURE_OPTIONS.64 += --bindir=/usr/bin/$(MACH64) CONFIGURE_OPTIONS.64 += --libdir=/usr/lib/$(MACH64) $(COMPONENT_SRC)/build-$(MACH32)/.configured: BITS=32 @@ -71,7 +71,7 @@ ($(RM) -rf $(@D) ; $(MKDIR) $(@D)) $(COMPONENT_PRE_CONFIGURE_ACTION) (cd $(@D) ; $(ENV) $(CONFIGURE_ENV) $(CONFIG_SHELL) \ ../configure $(CONFIGURE_OPTIONS) $(CONFIGURE_OPTIONS.$(BITS))) ../configure $(CONFIGURE_OPTIONS)) $(COMPONENT_POST_CONFIGURE_ACTION) $(TOUCH) $@
@@ -39,6 +39,7 @@ PKGFMT = /usr/bin/pkgfmt PKGMOGRIFY = /usr/bin/pkgmogrify PKGSEND = /usr/bin/pkgsend PKGLINT = /usr/bin/pkglint # Package headers should all pretty much follow the same format METADATA_TEMPLATE = $(WS_TOP)/transforms/manifest-metadata-template @@ -95,11 +96,14 @@ ifeq ($(IPS_PKG_NAME),) IPS_PKG_NAME = $(COMPONENT_NAME) endif ifeq ($(COMPONENT_SUMMARY),) COMPONENT_SUMMARY = $(COMPONENT_DESCRIPTION) endif IPS_COMPONENT_VERSION = $(COMPONENT_VERSION) .DEFAULT: publish .SECONDARY: $(GENERATED).fdeps .SECONDARY: publish: install $(COMPONENT_SRC)/.published @@ -152,24 +156,31 @@ cat $(CANONICAL_MANIFESTS) | $(PKGMOGRIFY) $(PKG_OPTIONS) /dev/fd/0 | \ sed -e '/^$$/d' -e '/^#.*$$/d' | sort -u | $(PKGFMT) >$@ $(MANIFEST_BASE)-%.compare: $(MANIFEST_BASE)-%.p5m $(MANIFEST_BASE)-%.compare: $(MANIFEST_BASE)-%.p5m canonical-manifests $(PKGMOGRIFY) $(PKG_OPTIONS) $(COMPARISON_TRANSFORMS) $< >$@ manifest-compare: $(COMBINED).compare $(GENERATED).compare @echo "Manifest comparison" @pkgdiff $(GENERATED).compare $(COMBINED).compare $(MANIFEST_BASE)-%.linted: $(MANIFEST_BASE)-%.resolved @echo "Linting $(@:$(MANIFEST_BASE)-%.linted=%) manifest" $(PKGLINT) $< $(TOUCH) $@ manifest-checks: manifest-compare $(PUBLISHED:%.published=%.linted) # mogrify the canonical manifest(s) # $(MANIFEST_BASE)-%.resolved: %.p5m manifest-compare $(MANIFEST_BASE)-%.resolved: %.p5m canonical-manifests $(PKGMOGRIFY) $(PKG_OPTIONS) $< $(PUBLISH_TRANSFORMS) >$@ $(MANIFEST_BASE)-%.published: $(MANIFEST_BASE)-%.resolved $(MANIFEST_BASE)-%.published: $(MANIFEST_BASE)-%.resolved manifest-checks $(PKGSEND) -s $(PKG_REPO) publish --fmri-in-manifest \ -d $(PROTO_DIR) -d . $< $(TOUCH) $@ $(COMPONENT_SRC)/.published: manifest-compare $(PUBLISHED) $(COMPONENT_SRC)/.published: $(PUBLISHED) $(TOUCH) $@ print-package-names: canonical-manifests @@ -182,7 +193,19 @@ $(PKGMOGRIFY) $(PKG_OPTIONS) /dev/fd/0 | \ sed -e '/^$$/d' -e '/^#.*$$/d' | sort -u canonical-manifests: $(CANONICAL_MANIFESTS) install-packages: publish @if [ $(IS_GLOBAL_ZONE) = 0 -o x$(ROOT) != x ]; then \ cat $(CANONICAL_MANIFESTS) $(WS_TOP)/transforms/print-paths | \ $(PKGMOGRIFY) $(PKG_OPTIONS) /dev/fd/0 | \ sed -e '/^$$/d' -e '/^#.*$$/d' -e 's;/;;' | sort -u | \ (cd $(PROTO_DIR) ; pfexec /bin/cpio -dump $(ROOT)) ; \ else ; \ echo "unsafe to install package(s) automatically" ; \ fi $(MOGRIFIED): install canonical-manifests: $(CANONICAL_MANIFESTS) Makefile $(PATCHES) ifeq ($(strip $(CANONICAL_MANIFESTS)),) # If there were no canonical manifests in the workspace, nothing will # be published and we should fail. A sample manifest can be generated @@ -200,4 +223,7 @@ @pkg search -H -l '<$(DEPENDS:%=% OR) /bin/true>' \ | sed -e 's/pkg:\/\(.*\)@.*/REQUIRED_PKGS += \1/g' >$@ pre-prep: required-pkgs.mk CLEAN_PATHS += required-pkgs.mk make-rules/prep.mk
@@ -36,12 +36,14 @@ $(GPATCH) -d $(@D) $(GPATCH_FLAGS) < $< $(TOUCH) $@ $(COMPONENT_ARCHIVE): $(COMPONENT_ARCHIVE): Makefile $(FETCH) --file $@ \ $(COMPONENT_ARCHIVE_URL:%=--url %) \ $(COMPONENT_ARCHIVE_HASH:%=--hash %) $(TOUCH) $@ $(COMPONENT_SRC)/.unpacked: $(COMPONENT_ARCHIVE) $(COMPONENT_SRC)/.unpacked: $(COMPONENT_ARCHIVE) Makefile $(PATCHES) $(RM) -r $(COMPONENT_SRC) $(UNPACK) $(UNPACK_ARGS) $(COMPONENT_ARCHIVE) $(TOUCH) $@
@@ -26,6 +26,9 @@ CONSOLIDATION = userland PUBLISHER = $(CONSOLIDATION)-build IS_GLOBAL_ZONE = $(shell /usr/sbin/zoneadm list | grep -c global) ROOT = / # get the most recent build number from the last mercurial tag LAST_HG_TAG = $(shell hg tags -q | head -2 | tail -1) LAST_BUILD_NUM = $(LAST_HG_TAG:build-%=%) @@ -35,6 +38,7 @@ BUILD_NUM = 0.$(shell expr $(LAST_BUILD_NUM) + 1) BUILD_VERSION = $(OS_VERSION)-$(BUILD_NUM) COMPILER = studio BITS = 32 PYTHON_VERSION = 2.6 transforms/manifest-metadata-template
@@ -28,6 +28,7 @@ set name=info.upstream_url value=$(COMPONENT_PROJECT_URL) set name=info.source_url value=$(COMPONENT_ARCHIVE_URL) set name=org.opensolaris.consolidation value=$(CONSOLIDATION) set variant.arch=$(ARCH) #license $(COPYRIGHT_FILE) license='$(COPYRIGHTS)'
@@ -28,7 +28,7 @@ <transform set -> edit value "\s$" ""> <transform set value="^\s*$" -> drop> <transform set name=info.classification value=".+:" -> drop> <transform set name=info.classification value=".+:\s*$" -> drop> # drop the license actions from the manifests, we aren't going to compare them <transform license -> drop>
