NetBSD Problem Report #50194

From dholland@netbsd.org  Thu Sep  3 07:38:31 2015
Return-Path: <dholland@netbsd.org>
Received: from mail.netbsd.org (mail.netbsd.org [149.20.53.66])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(Client CN "mail.netbsd.org", Issuer "Postmaster NetBSD.org" (verified OK))
	by mollari.NetBSD.org (Postfix) with ESMTPS id C36B5A6531
	for <gnats-bugs@gnats.NetBSD.org>; Thu,  3 Sep 2015 07:38:31 +0000 (UTC)
Message-Id: <20150903073831.831E214A142@mail.netbsd.org>
Date: Thu,  3 Sep 2015 07:38:31 +0000 (UTC)
From: dholland@netbsd.org
Reply-To: dholland@netbsd.org
To: gnats-bugs@gnats.NetBSD.org
Subject: teach make about zip files
X-Send-Pr-Version: 3.95

>Number:         50194
>Category:       bin
>Synopsis:       teach make about zip files
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    bin-bug-people
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Thu Sep 03 07:40:00 +0000 2015
>Last-Modified:  Sun Jun 29 21:45:00 +0000 2025
>Originator:     David A. Holland
>Release:        n/a
>Organization:
>Environment:
n/a
>Description:

Make has a dusty and disused feature where it can look inside ar files
in order to address individual members and write rules about them.
This allows, for example, updating single files within static
libraries after recompiling.

It would be fairly useful (e.g. for dealing with Java and jar files)
if make also knew how to look inside zip files. I don't think there's
any reason it can't use the same syntax and detect the archive type by
file header.

Note that it doesn't need to know how to extract or insert contents,
just read the directory information out, so the amount of code
involved will be small.

>How-To-Repeat:
n/a

>Fix:
n/a

>Audit-Trail:
From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/50194: teach make about zip files
Date: Thu, 3 Sep 2015 11:01:41 +0200

 On Thu, Sep 03, 2015 at 07:40:01AM +0000, dholland@netbsd.org wrote:
 > Make has a dusty and disused feature where it can look inside ar files
 > in order to address individual members and write rules about them.
 > This allows, for example, updating single files within static
 > libraries after recompiling.

 IMO this is a historic feature from the age where memory was extremely
 tight and doing anything on more than a single file was prohibitive. I
 don't think it should be extended and the general lack of equivalent
 functionality in most other build systems is a good indicator that it
 isn't all that useeful.

 Joerg

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc: 
Subject: Re: bin/50194: teach make about zip files
Date: Sun, 6 Sep 2015 00:03:22 +0000

 On Thu, Sep 03, 2015 at 09:05:00AM +0000, Joerg Sonnenberger wrote:
  >  On Thu, Sep 03, 2015 at 07:40:01AM +0000, dholland@netbsd.org wrote:
  >  > Make has a dusty and disused feature where it can look inside ar files
  >  > in order to address individual members and write rules about them.
  >  > This allows, for example, updating single files within static
  >  > libraries after recompiling.
  >  
  >  IMO this is a historic feature from the age where memory was extremely
  >  tight and doing anything on more than a single file was prohibitive. I
  >  don't think it should be extended and the general lack of equivalent
  >  functionality in most other build systems is a good indicator that it
  >  isn't all that useeful.

 I'm not convinced of that - it is dusty and disused for at least two
 other reasons; one being that ar is nearly useless, the other being
 that since the advent of shared libraries it doesn't make sense to do
 anything with single files in static libraries because that doesn't
 work for shared libraries.

 For jar files or the like (Java is not the only language environment
 to have hit on the notion of using zip files of compile results in
 place of linking) it can be used to avoid storing two copies of every
 build product on your disk; which is perhaps not that important given
 the cost of disk space, but it also doesn't cost very much to do.

 There's a second motivation though, which is that for any language
 where the interface definition of a module is a build product rather
 than an input, one wants make to be able to address the interface
 definition and the object file separately. If these come out as a
 single file, make needs to be able to look inside it; however, if
 these come out as two files, then we hit the problem where make chokes
 on multiple targets on the left of a rule.

 (This assumes that if you have

    foo.out: foo.src
 	$(COMPILE) foo.src
    bar.src: foo.out(foo.h)
 	$(COMPILE) bar.src

 that make will handle the dep through foo.out and foo.h properly; but
 if it doesn't currently it's fairly readily fixed, whereas the problem
 with multiple targets on the left is syntactic.)

 Granted it's a somewhat hypothetical situation but we could use it as a
 workaround for the base build problems with yacc output.

 -- 
 David A. Holland
 dholland@netbsd.org

From: Masao Uebayashi <uebayasi@gmail.com>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: bin/50194: teach make about zip files
Date: Sun, 6 Sep 2015 18:13:43 +0900

 I've been wondering if it's possible to rewrite multiple output rules
 using make(1)'s ``archive'' feature.  For example, timezone data is
 defined in a few files, and zic(8) outputs many binary files.  If
 those output files are archived, its rule can be written as something
 like this:

 tzdata.a: data files
         zic -d tmpdir ... # compile text definitions and generate
 binary zone files
         cd tmpdir && ar r ../tzdata.a */* # create tzdata.a archive

From: Roland Illig <roland.illig@gmx.de>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: bin/50194
Date: Sun, 29 Jun 2025 09:09:37 +0200

 I'm not convinced that zip files can solve the yacc issue. But even if
 so, it would already be possible with ar files, thus we wouldn't need
 zip support.

 I tried to understand how other programs would then work with the files
 inside the archive, but I didn't manage to understand it. Only very few
 programs are able to look inside ar files or zip files.

 In summary, I don't see any reason to support zip files in make.

 Roland

From: David Holland <dholland-bugs@netbsd.org>
To: gnats-bugs@netbsd.org
Cc: 
Subject: Re: bin/50194: teach make about zip files
Date: Sun, 29 Jun 2025 21:43:16 +0000

 On Sun, Jun 29, 2025 at 07:10:02AM +0000, Roland Illig via gnats wrote:
  >  I'm not convinced that zip files can solve the yacc issue. But even if
  >  so, it would already be possible with ar files, thus we wouldn't need
  >  zip support.

 IIRC the basic idea was something like

    foo.zip: foo.y
            yacc -d foo.y
            zip $@ y.tab.h y.tab.c
    foo.h: foo.zip
            unzip -d tmp foo.zip y.tab.h
            mv tmp/y.tab.h foo.h
    foo.c: foo.zip
            unzip -d tmp foo.zip y.tab.c
            mv tmp/y.tab.c foo.c

 Which does work, it's just messy and ugly and wastes cycles. IIRC
 around the time this PR got filed someone had suggested something like
 this.

 You could use ar instead; the problem with that is that storing
 anything other than .o files in ar isn't portable/doesn't work
 reliably/isn't recommended/whatever. (It worked for me when I tried it
 just now, but I also remember having had problems in the past.)

 The utility in this context of being able to look in the zip file to
 depend on the files inside it is pretty limited. What I wrote ten
 years ago kind of suggests I had something further in mind, but if so
 I've long forgotten what.

 However, I'm pretty sure the original motivation for filing this PR
 was trying to deal with jar files or possibly some boutique language's
 equivalent of them.

 In that context:

  >  I tried to understand how other programs would then work with the files
  >  inside the archive, but I didn't manage to understand it. Only very few
  >  programs are able to look inside ar files or zip files.

 That does, however, include the linking tools that use jar files and
 similar as input. So in particular you might reasonably write
 something like

    foo.class: foo.java
            javac foo.class
    lib.jar(foo.class): foo.class
            zip add lib.jar foo.class

 or even

    lib.jar(foo.class): foo.java
            javac foo.class
            zip add lib.jar foo.class
            rm -f foo.class

 (generalized over SRCS in any of the usual ways)

 Basically, a number of language environments now use zip files the way
 traditional Unix uses ar files. Therefore, it's reasonable to support
 looking inside zip files the same way make supports looking inside ar
 files.

 Note that this doesn't require being able to examine file contents so
 it doesn't require implementing anything complicated, just examining
 zip headers.

 -- 
 David A. Holland
 dholland@netbsd.org

Home	PR Database Search